I like the approach of running everything locally. I'm strongly of the opinion that the privacy angle for local models is going to keep getting stronger and more relevant. The amount of articles that come out about accidents happening because of people handing too much context to cloud models the more self reinforcing this will become.
It's only half of the solution though. If the models are trained in a closed way, they can prioritize values encoded during training even if that's not what you want (example: ask the open Chinese models about Tiananmen). It's not beyond imagining that these models would e.g. try to send your data to authorities or advertisers when their training says so, even if you run them locally.
So the full solution would be models trained in an open verifiable way and running locally.
Many Chinese models are being caught doing this (it's also required by law in China) but there was not much hassle.
Having said that Id easily trade some censorship about Chinese affairs I don't care about for the prudishness of American models. Though I generally get the abliterated versions of both.
Another angle is when you're passing untrusted content to the AI service, e.g. anything from using it to crawl websites to spam-detection on new forum user posts.
You can trigger the the service's ToS violation or worse, get tipped off to law enforcement for something you didn't even write.
local is best for privacy, but i personally think you don't need to go local.
anthropic, google, openai etc, decided that their consumer ai plans would not be private. partly to collect training data, the other half to employ moderators to review user activity for safety.
we trust that human moderators will not review and flag our icloud docs, onedrive or gmail, or aggregate such documents into training data for llms. it became the norm that an llm is somehow not private. it became a norm that you can't opt out of training, even on paid plans (see meta and google); or if you can opt out of training, you can't opt out of moderation.
cloud models with a zero retention privacy policy are private enough for almost everyone, the subscriptions, google search, ai search engines are either 'buying' your digital life or covering themselves for legal reasons.
you can and should have private cloud services, and if legal agreement is not enough, cryptographic attestation is already used in compute, with AWS nitro enclaves and other providers.
I personally think everyone should default to using local resources. Cloud resources should only be used for expansion and be relatively bursty rather than the default.
For about two years I experimented with writing local apps using local LLMs, but I often had to blend in a commercial web search API to make my little experiments useful.
I pay $13/month for Proton’s Lumo+ private chat LLM that contains an excellent built-in web search tool. I use it for everything non-technical, even just simple searching for local businesses, etc.
As an enthusiastic reader of books like Privacy is Power and Surveillance Capitalism, it feels good to have a private tool that is ready at hand.
do you have any provider recommendations? I've experimented with this on runpod serverless, but I've been meaning to dig deeper before I feel comfortable with personal data.
I saw a service named Phala, which claims to be actually no-knowledge to server side (I think). It was significantly more expensive, but interesting to see it's out there. My thought was escaping the data-collection-hungry consumer models was a big win.
> anthropic, google, openai etc, decided that their consumer ai plans would not be private. partly to collect training data, the other half to employ moderators to review user activity for safety.
That's two halves of "why", sure.
Another interesting half would be that those companies have US military officers on their boards, and LLMs are the ultimate voluntary data collection platform, even better trojan horses than smartphones.
Yet another "half" could be how much enterprise value might be found by datamining for a minute or two... may I suggest reading a couple of Martha Wells books.
The other thing, is encrypted inferencing a thing/service currently? I want to run my own models locally just because if I'm going to be chatting to it about my day to day life why send it to a server in plaintext.
> I like the approach of running everything locally. I'm strongly of the opinion that the privacy angle for local models is going to keep getting stronger and more relevant.
