I am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, browser history - maybe it even gets trained on your calls. Effectively an extension to your own brain.
I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI.
Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.
I also don’t want my AI in the cloud favoring a corporation’s values and goals.
But these tools are almost certainly going to come from them. And they want you to store all your data in the cloud too. And they want to keep you from running the software you want.
Eh. Patience. In the 70s, text editors, operating systems, and just about everything came from corporations, governments, and perhaps most commonly, universities when it came to software tools, and was hosted on their systems. As the hardware became more accessible, and the power needs diminished, it moved out from the mainframe to the client side.
This process will take time, but if we don't have the equivalent of chatGPT available on a home system (if not phone sized device) in 20 years, I'll eat my hat. There are already open source projects on Github that'll let you roll your own, but good luck coming up with the computing power.
And yet more recently everything began shifting to the cloud (storage, compute). Centralization leads to better economies of scale and therefore more efficiency.
It’s not clear to me this will ever change. While local compute will always improve (so local models become more viable), I believe that the largest and most capable models will always be cloud based.
If efficiency was the only metric, then I agree. However, we don't all need the largest and most capable models all the time. For example, stockfish running on my phone would destroy me and almost every human on Earth, even if every human collaborated together to try to beat it. Also, centralization isn't always the panacea people make it out to be. Google search was very efficient and centralized. They're about to have their lunch eaten by a start-up from nowhere. McDonald's hamburger is very efficient, but I eat that sparingly.
Varieties and different (inefficient) paths are still viable and usually needed to progress further. In many cases, centralization in the name of efficiency usually spells the beginning of the end for many firms (and countries).
I think there is one company very well placed to do this, Apple. Their revenue is driven by hardware sales, rather than advertising, and they have committed to some level of "privacy" focussed design (although obviously there are criticisms). On top of that their hardware is looking almost like it was design for exactly these use cases. Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs.
Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud.
They are looking for "the next big thing" that lets them sell hardware. Some people thinks that's going to be their VR/AR headset, I don't. It's going to be local AI running privately on your device. Putting the AI in the cloud doesn't help with hardware sales.
My only hesitation is that they have such a crap spell checker, surly you would solve first...
> Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs.
Yes, university research from 2011 already showed a privacy focused design to machine learning[1]. But will this be private or public like Wikipedia, Bitcoin and Bittorrent?
> Effectively an extension to your own brain.
Correct. After 1996 fantasy/visionary publications about this, we now know roughly how to do this. Most difficult problem is who should own it, no corporation in control is desired. That means full decentralisation, federated learning is not enough.
But fully decentralised learning is hard. Try to apply machine learning in permissionless, byzantine, unsupervised, decentralised, adversarial, continuous learning context. See my lab at Delft University focused for a decade already on "The Global Brain"[2]. With 2.3 million download and crowd-sourcing, we might get there..
I don't think iCloud is a great example, because there's a definite value add in having your files synced between multiple devices and your computer. A better example might be Siri, which does basically nothing without an internet connection.
iCloud provides a lot of value, because storage is extremely expensive on all iDevices, and you can't upgrade it. Also Syncthing can be used for a synchronization between devices, if you control them.
I think Apple will be hesitant to release an AI on their platforms that can be made to say unpredictably bad things. For now as far as I have seen there’s only censoring and that’ll only lead to a game of people trying to work around it.
I remember google desktop search. It was amazing, even today's win 11 search doesnt come close. So something snappy like that with LLM will be perfect.
Yes, I whole-heartedly agree. What we need is a proliferation of AI tools. Centralization of compute in the hands of a few corporations could be worse than monoculture farming. If we don't create, mix, and match our own models we're offloading all decisions to opaque companies like the "open" OpenAI.
It's probably the future you want. But fact is, most of peoples emails, documents, browser history, travel history, bookmarks, contacts, etc., already live in the cloud. People are obviously not concerned with it, so I would say the future of LLM's is in the cloud using all of the cloud data.
Computers are becoming smaller and the internet is becoming even more ubiquous. I don't see your future happening.
Why would I spend a lot of money, space, and maintenance time for a computer with a good GPU when I can just use any of the two internet connections I'm already paying for? I'm already trusting corporations to read my email, and no ChatGPT answer can get more personal than that.
This is also why I think cloud gaming is only going to become more popular.
There's a stronger resistance to cloud than ever before after all the leaks, hacks and pushed agendas.
These models aren't cheap to host and they'll lose to free local models that don't have boundaries or agendas. Cloud AIs inevitably will try and monetise by selling you crap or be funded to shape your opinions.
> Cloud gaming was a crash and burn story already.
What? No it isn’t. One company failing doesn’t mean the industry has crashed and burned. I use Xbox cloud gaming all the time along with millions of other people.
Even Microsoft has admitted (between the lines) that cloud gaming isn't as profitable as just selling games. And it's not just one company that's failed either. Google, Onlive, Gaikai, there was a company that made a Cloud console a couple years back, the list goes on. Besides, the latency will never be low enough, and the bandwidth cheap enough, for a lot of people to make it a viable solution.
It is, but AI is a different kettle of fish altogether.
We already push back on targeted advertising and tracking, think what AI is capable of with your data. If its writing your emails, if its summarising for you, if its consulting for you...
It can push agendas and influence. Its quite dangerous.
These models need to be local as much as possible and they need to be better than what the cloud can offer.
At least for me this isnt one the decentralised Internet can take if we don't want to go down the dystopian future route.
Why would this be a risk? The data is already on your machine. It would be much easier for a hacker to target that data directly, as they do today, than to try to extract some impossibly obfuscated form of it from an LLM.
Well if they are interested in data. If they are interested in asking a prompt "hey what's my cc number and password again?" That'd bypass an awful lot of encryption. Would also be easy to find blackmail for a lot of people thinking it's a secure way to lookup crimes.
These things will inevitably be included with the OS soon
Desktop, mobile, even consoles - especially video game consoles where games can tap into the shared resource for their NPCs instead of bundling it in their package
GPU based token generation will be good enough in speed
I was pretty impressed by the power and accuracy of the Google Translate offline models (which predate some of the amazing new LLMs), also in a seemingly amazingly small space—tens of megabytes per language, if I remember correctly?
Firefox's builtin translation is not bad either. I've seen it make a mistake here and there, or fail to translate certain words and expressions but it works generally well.
"The Firefox Translations extension can automatically translate content from the web pages you visit. Unlike some cloud-based alternatives, this extension translates text locally in Firefox, so the content you’re translating doesn’t leave your machine."
I feel the same way with running stable diffusion on my MacBook (DiffuseBee app). The idea that I can generate an image of almost anything and it has references to what it means in the real world, and carry that in my backpack, is little hard to comprehend.
Given the biased nature of many if not most Wikipedia articles it would be certain that such a system would bend the truth to an unacceptable degree. Train the thing on more than just Wikipedia, e.g add Encyclopedia Britannica to the training set. This is the one area where "diversity" really matters: diversity of opinion. Using a single biased source as your oracle will turn you into a pawn for those who control the bias.
At the risk of conflict of interest, there's a wikipedia page that acknowledges that niche sections of the site have biases. Of particular interest is the section on Croatia.
It baffles me how you can either have a news/information source with a left-wing bias or a total cesspool of untrustworthy information with a glaringly obvious agenda.
This comment seems to get some hate, but all you have to do is skim over the linked Wikipedia article to verify what I'm saying.
I agree, and for another demonstration, I invite everyone to look at similar sites as Wikipedia. They are either "left wing bias" (RationalWiki) or one with an agenda, divorced from reality (Conservapedia). It almost looks like that if you organize basic democratic cooperation among people, that's automatically "left wing" from where our current narrative stands.
> They are either "left wing bias" (RationalWiki) or one with an agenda, divorced from reality (Conservapedia).
#/bin/env bash
read -p "Please enter your personal bias (e.g. conservative, liberal, progressive, etc.) philisophy: " bias
echo "they are either ${bias}ly biased or with an agenda, divorced from reality."
While it was not said out loud, it reminded me of the "both sides" fallacy. Like how schools should teach both sides, creationism and evolution, as if both have the same merit. Likewise, the bash script implies that bias is bias.
Do you think the same about OP's statement that I reflected on, stating "you can either have a news/information source with a left-wing bias or a total cesspool of untrustworthy information with a glaringly obvious agenda"?
As a point of reference, if you search high traffic websites with "Left bias" and also with "Right bias"[0], it will turn out that the average factuality of the sources are higher on the "Left bias" news sources.
This makes me think that the point of reference, the "center", is now off-center, to the right. To the point where actual reality seems "left". "Facts Have a Well-Known Liberal Bias". Considering this, sources even further to the right will of course have worse information, same as they would on the far left. And this is for news sources - what I added was that I observed the same for wikis.
Wikipedia is as biased as the writers of the articles. Wikipedia has articles outlining bias in its own articles. These things are authored by people, and people are biased. Wikipedia knows this and so should its readers. They aren't unique.
Since the first article hit the site which clearly showed off the personal bias of the author. Then since the first article was edited by a group of like-minded editors who approved of each others edits but were vigilant in keeping other opinions off the page. These groups of editors come in different flavours ranging from ideologically driven [1] - these control the narrative on pages related to politics and politically sensitive topics like SARS2 and "Climate" - to commercially employed to keep unwanted narratives off certain pages.
> and why EB isn't?
Who says that Encyclopedia Britannica is not biased? They are to a certain (although lesser) extent, they just happen to be differently biased. That is where my remark on the value of diversity of opinion comes in.
US politics, definitely. But also any other topic that attracts an internet mob large enough to infiltrate the editor community.
Take the first few paragraphs of Donald Trump [0] vs. Joe Biden [1] and try to compare the way they're presented:
Donald John Trump [...] and his businesses have been involved in more than 4,000 state and federal legal actions, including six business bankruptcies. Trump promoted conspiracy theories and made many false and misleading statements during his campaigns and presidency, to a degree unprecedented in American politics. Many of his comments and actions have been characterized as racially charged or racist and many as misogynistic. [...] He reacted slowly to the COVID-19 pandemic, ignored or contradicted many recommendations from health officials, used political pressure to interfere with testing efforts, and spread misinformation about unproven treatments.
There's not a single positive sentence about him. He must have done something positive in his 4 years of reign, but Wikipedia doesn't tell me that in an objective manner. Meanwhile...
Joseph Robinette Biden Jr. [...] has addressed the COVID-19 pandemic and subsequent recession. He signed the American Rescue Plan Act, the bipartisan Infrastructure Investment and Jobs Act, the Inflation Reduction Act, the CHIPS and Science Act, and the Respect for Marriage Act, which codified protections for same-sex marriage and repealed the Defense of Marriage Act. He appointed Ketanji Brown Jackson to the Supreme Court. In foreign policy, Biden restored America's membership in the Paris Agreement on climate change.
There's not a single negative sentence about him. He must have done something negative in his 4 years of reign, but Wikipedia doesn't tell me that in an objective manner.
Whenever someone tries to portray someone as completely evil or completely saint-like, I call bullshit.
i know you're being sarcastic, but do i think its interesting that centrist politics is sometimes viewed as less corrupt just because the bias is harder to articulate
Your [...] are very convenient, because you skipped over the part in paragraph 3 of Trump's article where they go over some of his main "achievements", which are presented neutrally and some people might consider positive:
[Trump] signed the Tax Cuts and Jobs Act of 2017, which cut taxes for individuals and businesses and rescinded the individual health insurance mandate penalty of the Affordable Care Act. He appointed 54 federal appellate judges and three U.S. Supreme Court justices. He initiated a trade war with China and withdrew the U.S. from the proposed Trans-Pacific Partnership trade agreement, the Paris Agreement on climate change, and the Iran nuclear deal.
and you also cut out a mention of a pretty significant Biden failure (in outcome, at least):
[Biden] oversaw the withdrawal of U.S. troops from Afghanistan that had been negotiated and begun under the previous administration, ending the war in Afghanistan, during which the Afghan government collapsed and the Taliban seized control.
But moreover, the introductory paragraphs of a Wikipedia article, like the introduction to most pieces of writing, are meant to establish an overview. Assuming that the reader reads nothing else, what basic idea should they have of the thing?
Donald Trump spent his four years in office much like he spent most of his previous business career: lying and committing various forms of fraud. While president, just for variety, he added lying and committing fraud about COVID, and, oh yes, inciting a violent insurrection to overthrow the American government. That should be the first thing you take away. He may have accidentally done some positive things, but none of them make the lead paragraph, because they won't be in the top 3-5 things that history remembers him for. "Scholars and historians rank Trump as one of the worst presidents in American history" appears near the end of the introduction (right before jumping back into more recent events), because that's a germane summary.
Joe Biden has spent his 2+ years in office being a relatively uninteresting mainstream-to-slightly-progressive Democrat. That should be the first thing you take away. He's not perfect, but he's doing normal president stuff. Once he's no longer president, no doubt his article will be updated to look like Obama's, which currently ends the introduction with:
Rankings by scholars and historians, in which [Obama] has been featured since 2010, place him in the middle to upper tier of American presidents.
> Donald Trump spent his four years in office much like he spent most of his previous business career: lying and committing various forms of fraud. While president, just for variety, he added lying and committing fraud about COVID, and, oh yes, inciting a violent insurrection to overthrow the American government.
Interesting that I can't find any reference for such claims in the Wikipedia article. The whole introductory paragraph contains the total of 2 sources - one a politico.com article [0] and the other a Siena College Research Institute post [1] - neither of which provide any claims other than that Trump was unpopular and "rated as worst".
You can't with a straight face claim that this is unbiased writing.
This is actually a pretty sound summary of the two. The one guy just got charged for rape, recently another one of his companies went down for fraud, he invited a riot on j6 to overthrow democracy, is under investigation for stealing hundreds of classified documents, lied tens of thousands of times(people take the time to count these things with references for presidents), issued dozens of legal challenges to try to overthrow democracy all of which were found to be based on nothing across the country, called racists dressed up as Nazis fine people, defended adversarial dictators, ate classified materials, national archives had to pull some out of toilets, etc.
The other guy mostly eats ice cream and hangs out in delaware... They tried investigating him already a few times and his son and came up empty handed. Maybe time will prove he's not the best guy ever, but it's pretty safe to say the other guy should be in jail because he's a prolific criminal.
> i have no clue why everyone isn't talking about this all the time.
Talking maybe, not doing it because it's still too out of reach for most of us? I've got a laptop with 32 GB RAM and a 2GB NVIDIA card. At which speed can I run that model, if I can run it at all?
2gb vram is not enough for running LLM but you can run on CPU gpt4all provides gui just click click download model click and you are ready, just give it a try
2GB VRAM means you can run things comparable to GPT2, a glorified Markov chain. On your CPU you could run much larger models at far from real time speeds.
I currently have a copy of offline Wikipedia (and Stack Overflow, WikiVoyage, etc.) via Kiwix on my laptop and also LLaMA-style models at the same time (since I like things being local), and with respect to asking an LLM questions, it would be great if it could leverage the offline Wikipedias to ensure it stays grounded and not make up facts.
The magic of swap means you can leave those applications open and as long as they're idle they'll just quietly wait for you on disk until you've finished your LLM adventure.
On many modern systems swap is quite small and not large enough to hold the entire system. For a 32GB machine, Ubuntu recommends 6GB and redhat recommends 6.4GB (20%). In a VM that number is much much lower. Besides, some libraries allocate memory in such way that you get artificial memory spikes in physical memory.
I know this because I spent this weekend debugging crashes in an LLM pipeline :(
I don't know about every time but honestly the fact that it'll just fill in the gaps when it doesn't know means that you have to treat all responses with a bit of suspicion.
Or an Encyclopedia Britannica on the shelf/in pdf. People don't know that they need it, but it indeed feels amazing when you need an information, and it is there.
Now I have an excuse to purchase high powered machines. As a programmer who doesn't play games, edit video, or do anything requiring a high-powered graphics card, I always eyed high-end rigs as a wasteful indulgence. Not anymore! Now I can't wait to train my own models on my own data and make the computer do things for me, like its 1982 all over again.
The thing about transformer models that are impressive is that they compress all sorts of stuff about human understanding down into a incredibly tiny space. They asked GPT-4 to explain GPT-2 and it found things like "something done right" or "metaphors" corresponded to neurons. It's this meta understanding of thought, just like diffusion models are this meta understanding of art. The real innovation is one can turn a crank and it keeps getting deeper and more subtle insights into the whole thing. If you turn that crank with A100s for a few days it gets very deep and doesn't hit a wall. In the earlier days of machine learning that didn't used to happen. Understanding was bounded at some point and didn't increase, but it seems transformers just keep going.
I think this will be more mainstream if someone create an installable app (win/mac/linux), right now I did a bit of research and I have to install several stuff (with https://huggingface.co/), it was confusing to a non advance user, I wasn't able to do it.
Within a few minutes you can sign up to OpenAI and see the potential of ChatGPT immediately, in you own hands. The potential of crypto/web3 was always an abstract idea (jumped on by so many scammers), there was nothing you could actually use to see the utility.
Plus there is tooling like Copilot that works and is useful. I would love to have something like copilot integrated across my whole system, gathering all my current context, like terminal, notes, mail, issue tracker and have even just plain, simple autocomplete everywhere. It is feasible, and I'm pretty sure we will have more and more integration. I was sceptical, but copilot changed my mind.
It seems like everyone in our company uses generative AI at the moment: marketing department coming up with ideas, designers creating illustrations, programmers writing code etc. I.e. there's a lot of useful real life application of generative AI right now, people actually use it for work. Crypto, for most part, in my opinion, did not provide as much value to the average person. And it's now possible to run LLMs locally, so you shouldn't be afraid of exit scams :)
I would pay several hundred dollars a month to keep access to GPT-4, and never found any use at all for crypto. There might be hype, but there’s also an insane amount of utility
For almost anything, I find it a better Google, and I can go fact-check it afterwards.
It’s a better StackOverflow and better than most project or dev docs for me: I’m an experienced developer who’s writing Python for the first time. I know what I want to achieve each time (how do I produce the Typing type for…) but I find it much easier to use GPT.
I use it to search for books or films or video games where I half remember the details, and it gives me options.
I recently finished a dissertation (on GPT, as it turns out) and it was useful for me to research sensible correlation types I could use and build out my understanding (“wait, is Spearman’s rank is just Pearson in the rank order?”). Also I could copy and paste sentences I didn’t understand about methodologies in papers I was reading, or ask it abstract questions like “is there a standard name for a modified Likert scale but with a ‘not enough data’ option”.
I was really struggling to work out how to change just one page orientation in Word, on a Mac. First few Google results I had talked about the Windows version, which had features mine didn’t have. GPT put me onto section breaks.
Was Caffè Nero named after the emperor, checking my intuition about how short-term SSH auth certs work, jq recipes, why were Man U so popular when I was a kid, why did they suddenly start winning more often?
I am curious about a lot of things, and GPT provides me instant answers without having to click through bullshit. If it’s something it needs to be right about I can double-check it’s answers. I am averaging ten topics a day on it, most have a bunch of follow-up questions.
LLM has provided real value for me the last couple of months, both as amusement, as a way to "talk" while working from home and by solving problems.
And I was late to the party and arrived skeptical.
With crypto I arrived early (2009) and in hindsight realized that the only real value I can see from most of its existence is transfer of money to and from hard-to-reach-financially places and as an object of financial speculation.
Yes, this. Once they get a smidge better and can answer questions? Every student will have the King's Tutor - it will be Blooms Two Sigma in action.
...which will drive down the cost of all knowledge work and rather than reconfigure our society we'll simply have no entry level jobs anymore and the few people who still have jobs will still be expected to work full time but have insane outputs.
You had all that knowledge before too, free from the internet. It’s just easier and common thing to do to live within the system with all the other people. But this is changing also with the changing world conditions.
I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI.
Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.