I love Yandex. They are the best search engine by far for politically controversial topics. They also release a language model to benefit everyone even if it says politically incorrect stuff. They also name their projects "cocaine" probably to perhaps to prevent western competitors from using them.
You look at OpenAI and how they don't release their models mainly because they fear "bad people" will use them for "bad stuff." This is the trend in the west. Technology is too powerful, we must control it! Russia is like... Hey, we are the bad guys you're talking about so who are we keeping this technology from? The west has bigger language models than we do, so who cares. Also their attitude to copyright and patents, etc. They don't care because that's not how their economy makes money. Cory Doctorow's end of general purpose computing[1] and locked down everything is very fast approaching. I'm glad the Russians are around and aren't very interested in that project.
Search Google and Yandex for "2020 election fraud." The results are VERY different. The Zach Vorhies leak shows that Google regularly does blatant censorship for political purposes.[1]
Totally, just like how if you want to find out what really happened in Tiananmen Square in 1989, your best bet is Baidu. Totally different results than what Google gives you!
I sincerely have deep respect for Yandex for releasing this, and Baidu for some of the amazing research they've released over the years, but both are deeply deeply beholden to their local governments in a way that is incomparable to the relationship between Google and the US government.
Remember that the NSA was literally digging up and tapping fiber around Google data centers in a secret program called MUSCULAR because they didn't think Google was being cooperative enough when handing over data that they were requesting.
Google: 118M results. Top link is the best resource on verified election fraud cases.
Yandex: 9M results. The top two links are pretty suspect. Top link promotes Dinesh D'Souza's 2000 Mules documentary in the banner which at best is a one-sided take on election fraud. At worst, very misleading.
This is a weird comment because yes, that's exactly what the above person was saying. It shows results that google won't give you.
Secondly, I've yet to see any criticisms of 2000 mules data that aren't addressed by the stringency in the analysis they claim to have done.
I thought the information they presented was extremely valuable. Are we going to overturn an election at this point? No. But the vulnerabilities to the mail-in ballots were obvious, then lied about, then ignored, then clearly taken advantage of. I want to live in a democracy because voting matters. I especially don't want NPO's destroying this by taking advantage of flawed voting infrastructure.
If there are legitimate criticisms of the methods, the data, or anything else coming out of this film I expect a legitimate presentation that can break it down using the actual data in question. All I've seen so far is shilling and gas-lighting.
If you have any evidence, any evidence at all, of significant mail-in ballots fraud, then you should write it up and publish it; and even present it to the USDOJ, because you would have succeeded where Trump's highly-paid teams of lawyers failed.
It's not about whether it happened or not, it's whether it's reported or not.
I personally believe (with obviously no proof) there was definitely fraud going on, on both sides. With such an archaic system and such a great economic and power incentive, you would be stupid not to do it. For sure mail in ballots made it even easier than in the past.
I've heard about Russian hacking the elections after Trump won for a good 2 years.
Why should 2000 Mules be promoted in results at all? It's total BS, it's progenitor is a criminal that's was guilty of Election Campaign Finance Fraud.
I don't know man, "thegatewaypundit.com" as a top reputable source? seems to me like it's not "honest two-sided results" but just, well, a rather random mix of result of widely varying quality. Mad Altavista vibes!
What I'm trying to say is that even if you believe that "was the 2020 US election stolen?" is worth debating, which it isn't, the yandex results are shit.
If you get all your information through mainstream channels, and you don't want to see anything contradicting those channels then you should continue to use Google because they explicitly implement the algorithms on controversial topics to prefer mainstream news sources[1]. What I mean by "better" in terms of controversial searches is that on controversial matters, it will rank the searches the same way it does for all other searches. I mean yeah, I don't have access to the internal code base of Yandex, but it certainly feels more organic.
Btw Wikipedia’s first few sentences on Breitbart are not inspiring
> Its journalists are widely considered to be ideologically driven, and much of its content has been called misogynistic, xenophobic, and racist by liberals and traditional conservatives alike.[10] The site has published a number of conspiracy theories[11][12] and intentionally misleading stories.[13][14]
This is the association fallacy, which is, unfortunately, how most people determine what to believe these days.
An absurd example of this fallacy would be, Wikipedia, which you cite, has articles that indicate tobacco smoking may cause disease. The nazis were also anti-smoking[1]. Therefore Wikipedia is Nazi propaganda and you should not trust anything on there.
It is not the association fallacy; the role of a news site is to provide news, which includes fact-checking the work of their "journalists."
If Breitbart pulled a Fox News and argued in court that their goal was to entertain and not inform, then you have a point! But until then, you have a terrible misunderstanding of journalistic integrity and what it means for a publisher to attach their name to a journalist's work.
> This is the trend in the west. Technology is too powerful, we must control it!
I take it that you're either too young or too untraveled to be aware of the level of state control of technology in "the east". Xerographic machines, mimeographs, and other similar reprographic devices used to be highly controlled machinery behind the Iron Curtain. This is absolutely not something exclusive or even peculiar to "the west".
>> They are the best search engine by far for politically controversial topics
FYI, they are Russian subject that follows ALL their censorship laws (and oh boy do they have a lot of it).
>> probably to perhaps to prevent western competitors from using them
The irony here. All yandex products are exact copies of western, adjusted to local market.
Actually they're not, some of the Yandex products are actually better and pretty innovative (ignoring the political stuff). Maps and Go are especially good. Ditto with Russian banking apps, they put American bank apps to shame.
At the same time, from a consumer perspective, this sacrifice to internal political pressures won't hamper your usage of the product. It's not like if international borders were the most important annotation on a map for everyday use.
Edit: in fact, I just checked, yandex maps still shows state borders.
It doesn't distinguish state and region borders anymore, like it did before. No borders on the overview map either. Just zoomed into a random place on the map, and I can tell where Turkey ends and Bulgaria begins only because the city names are different.
I didn't see it as I was looking at France. It's weird, because at large scale there are no borders, at medium scale there is a weird mix of national and local borders (Western EU countries have state-level borders, RU/BY/UA/US/CN have local borders, ...).
And to take your example, I have to zoom quite close for BG/TR to switch from state-wide to local borders.
> it's a Russian company breaking Russian laws and getting away with it
I don't think you've lived in Russia if you need to ask that question. Breaking the law and getting away with it is a way of life in Russia, that goes for all institutions and social strata
Breaking random laws? Sure. Breaking laws specifically made to enable central government control of independent media? Uh, how do you do that? Have you not noticed for how minor things regarding freedom of expression have been Russian people getting into jail recently?
By being on the internet. Russia has always been good at literally hitting you with a physical club if you're crazy enough to take a sign to the streets, but the Russian state doesn't understand the internet, or really anything that's sort of underground or intangible.
There's a reason the country is probably the world's largest place for all things piracy related, scihub and so on. It's not just laxer IP laws, it's also that tech in particular has always skirted all kinds of regulation freely, it's why the country has a relatively healthy tech industry despite at times suffocating regulation. The prevalence of cybercrime in the country is another example of it. Being censorious doesn't make you competent.
They literally have blocklist of sites that kremlin doesnt like and it acts somehow similar to yandex news in this part. The difference here is more that google filters stuff for usa and yandex for russia
This is one of the funniest threads I’ve ever seen on this website. People are yelling at eachother about the CIA and the legitimacy of Israel and Assange and the definition of fascism and… anything that pisses anybody off about international politics in general. In a thread about a piece of software that’s (to me and likely many others) prohibitively expensive to play around with.
Anyway I hope somebody creates a playground with this so I can make a computer write a fan fiction about Kirby and Solid Snake trying to raise a human baby on a yacht in the Caspian Sea or whatever other thing people will actually use this for.
To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.
There are tons of commercial uses for these models. I've been experimenting with an app targeted toward language learners [1]. We use large language models to:
- Generate vocabulary - e.g. for biking: handlebars, pedals, shifters, etc
- Generate translation exercises for given topic a learner wants to learn about - e.g. I raised the seat on my bike
- Generate questions for the user - e.g. What are the different types of biking?
- Provide more fluent ways to say things - I went on my bike to the store -> I rode my bike to the store
- Provide explanations of the difference in meaning between two words
And we have fine tuned smaller models to do other thing like grammar correction, exercise grading, and embedded search.
These models are going to completely change the field of education in my opinion.
I started to work on something similar but way behind your project. I really believe AI models can help us as humans learn better! Do you have a blog or any other writeups on how you approached these problems?
We're using these at where I work (large retail site) to help make filler text on generated articles. Think the summary blurb no one reads at the top. As for why we're writing these articles (we have a paid team that writes them too), the answer is SEO. This is probably the only thing I've seen done with a text model in production usage. I'm not 100% sure what model they're using.
Yeah I'm not a huge fan of it. I'll never forget the look in our UX person's eyes when she realized that our team doesn't exist to make customer's experience better (there's a ton of other teams for that) but to make Googlebot's experience better. Right now we're in the process of getting publishers you've heard of to write blurbs for best lists, but we're supplying the products so it's not really a best list.
I can't say I'm a big fan but my teams is great and I don't have time to look for a job right now.
I hate this so much. These tools are getting better, so often you realise only half way through that you are reading AI text. Then you have to flush your brain and take a mental note, to never visit that site again.
There's a line in one of Douglas Adams' books where he says something along the lines that things like VCRs were invented to watch TV programs so that you don't have to.
Who would have thought that one man's joke would become a reality?
"worthless" huh, not everyone can afford inference of a ~500gb models, depending on the the speed/rate you need you might definitely go for smaller model
But maybe your sentence was more about "after BigScience model, open-sourcing anything smaller than that will be useless" which isn't necessarily true either, because there is still room to improve parameter efficiency, i.e. smaller models with comparabale performances
It kinda seems like a model trained on multiple languages would to some extent be better at English than a model trained only on English? I mean so much of English comes from other languages, and understanding language as a concept transcends any specific language. Of course there are limits and it needs good English vocabulary and understanding, but I feel the extra languages would help rather than hinder English performance.
My guess is they're mostly vanity projects for large tech companies. While the models have some value, they also serve as interesting research projects and help them attract ML talent to work on more profitable models like ad-targeting.
They did not publish benchmarks about quality of the models, which is very suspicious.
I personally squinted hard when they said removing dropout improves training speed (which is in iterations per second), but said nothing about how it affects the performance (rate of mistakes in inference) of the trained model.
I agree that the lack of benchmarks makes it hard to determine how valuable this model is. But on the topic of dropout, dropout has been dropped for the pretraining stage of several other large models. Off the top of my head: GPT-J-6B, GPT-NeoX-20B, and T5-1.1/LM.
An equally plausible frame is that once a technology becomes replicated across several companies, it makes sense to open source it since the marginal competitive advantage are the possible resultant external network effects.
I don't know if that's the right way to think about the open sourcing of large language models. I just think we really can't read too much into such releases regarding their motivation.
From what I've seen, using these huge models for inference at any kind of scale is expensive enough that it's difficult to find a business case that justifies the compute cost.
Those models aren't trained with the objective of being deployed in production.
They are trained to be used as teachers during distillation into smaller models that fit the cost/latency requirements for whatever scenario those big companies have. That's where the real value is.
I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training.
Now, that's not your point - cost != price. But, still...
I can't think of anyone having a few hundred GPUs around unless:
- They were into Ethereum mining and quit.
- They've already built a cluster with them (e.g. in an academic setting).
- They live in a datacenter.
- They are a total psychopath.
But even assuming one magically has all those GPUs available and ready to train, I don't want to calculate the power cost of it anyway. Unless one has access to free or extremely cheap electricity it would still be very expensive.
Only half-joking use case: active communities like this one on HN make sites attractive to human visitors. A new site could use bots to fake activity. Not sure it would work in the long run though.
I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of that will be, especially if the models improve drastically. With so much of our social lives being moved online, if we have the easy ability to create fake lives of fake people one has to wonder what's real and what isn't.
The bots/machine vs human reminds me of that famous experiment from the 30s in which Winthrop Kellogg[0], a comparative psychologist, and his wife decided to raise their human baby (Donald) simultaneously with a chimpanzee baby (Gua) in an effort to "humanize the ape". It was set out to last 5 years but was relatively quickly abrupted after only 9 months. The explicit reason wasn't stated only that it successfully proved the hereditary limits within the "nature vs nurture" debate of a chimpanzee, the reticent statement reads as follows:
>Gua, treated as a human child, behaved like a human child except when the structure of her body and brain prevented her. This being shown, the experiment was discontinued
There have been a lot of speculation as to other reasons of ending the experiment so prematurely. Maybe exhaustion. One thing which seemed to dawn on the parents - if one reads carefully - is that a human baby is far superior at imitating than the chimpanzee baby, frighteningly so, that they decided to abort the experiment early on in order to prevent any irreversible damage in the development to their human child which at that point had become far more similar to the chimpanzee than the chimpanzee to the human.
So, I would rephrase "the internet is dead" into "the internet becomes increasingly undead" because humans condition themselves in a far more accelerated way to behave like bots than bots are potentially able to do.
From the wrong side this could be seen as progress when in fact it's opposite progress. It sure feels like that way for a lot of of people and is a crucial reciprocal element often overlooked/underplayed (mostly in a benign effort to reduce unnecessary complexities) when analyzing human behaviour in interactions with the environment.
Case in point: recently, I've noticed that I'm getting more and more emails with the sign off "Warm regards." This is not a coincidence. It is an autosuggestion from Google. If you start signing off an email, it will automatically suggest "Warm regards." It just appears there -- probably an idea generated from an AI network. There are more and more of these algorithmic "suggestions" appearing every day, in more and more contexts. This is true for many text messaging programs: There are "common" replies suggested. How often do people just click on one of the suggested replies, as opposed to writing their own? These suggestions push us into conforming to the expectations of the algorithm, which then reinforces those expectations, creating a cycle of further pushing us into the language use patterns generated by software -- as opposed to idiosyncratic language created by a human mind.
In other words, people are already behaving like bots; and we're building more and more software to encourage such behavior.
Those suggestions appear in Google chat too and even if you don't click on them, the simple fact of reading the suggestion makes you much more likely to type it yourself. There's clearly a priming effect to it.
Over my career I've worked with:
- Doctors
- Lawyers
- Engineers
- Fund managers
- Academics (hard and soft sciences)
- Mentalists/Hypnotists
All of them believed that they're specific training and temperament made them immune from simple persuasion techniques and that they were purely rational actors.
None of them struck me as any more rational/more independent thinkers than anyone else off the street
It is typical to rate yourself above your actual self.
Even when someone rates oneself down like when saying of themself that they're dumb, ugly or whatever, they generally mean it in a lesser fashion than for any other peer they'd attribute as such.
But its not above, its ascribung a mythical ability that does not exist - we don't talk about people who think they are psycic as optimistic, we call them crazy.
these guys are similar, except it's common belief.
Which is why it's important for folks to start applying AI to more interesting (but harder, more nuanced) problems. Instead of making it easier for people to write emails, or targeting ads, it should be used to help doctors, surgeons and scientists.
The problem is that these problems are less profitable. And that the companies with enough compute to train these types of models are concerned about getting more eyeballs, not making the world a better place.
The problem is not that those problems are less profitable. The problem is a combination of
1. Those problems are much harder
2. The potential harm from getting them wrong is much larger
Yup, I definitely agree that they're harder (and noted this). But I'm not sure I agree with your second point. Or rather, I think there's some nuance to it.
Sure, using AI to treat people without a human in the loop would clearly do harm. But using AI as an assistant, to help a doctor make the right diagnosis, seems like it'd do the opposite. It'd help doctors serve a larger patient population, make less mistakes, and probably equate to less harm in the long run.
Anyway, I think we can all agree that using AI for anything other than ad targeting is a net win.
I don’t know if this is how it still works, but early attempts were modeled as classification problems with hundreds of hand picked completions. Can’t predict something really bad if it isn’t in your prediction list. This limits the surface of bad things to cases of tone mismatch like “sounds great” when talking about someone grieving a loss or something.
Doesn't GMail collect the data in some form of federated learning nowadays, like GBoard does? Federated learning does seem to be able to create the unintended positive feedback loop, converging on a single phrase and causing the users to lock themselves in a bubble.
Actors attempt to imitate humans. “Good acting” is convincing; the audience believes the actor is giving a reasonable response to the portrayed situation.
But the audience is also trying to imitate the actors to some degree. Like you point out, humans imitate. For some subset of the population, I’d imagine the majority of social situations they are exposed to, and the responses to situations they observe, are portrayed by actors.
At what point are actors defining the social responses that they then try to imitate? In other words, at what point does acting beget acting and how much of our daily social interactions actually are driven by actors? And is this world of actors creating artificial social responses substantially different than bots doing the same?
This is a common phenomena where the fake is more believable than the real thing due to over exposure of the imitation.
Famously the bald eagle sounds nothing like it does in tv and the movies and explosions are rarely massive fireballs. For human interaction it’s much harder to pin down cause and effect but if it happens in other cases it would be very surprising to not happen there.
Someone wrote once about how Wall Street people started behaving like the slick image projected of them in movies in the 80s, namely of Michael Douglas; before that they were more like the "boring accountant" type.
It's the commonly believed reason; the child starting to take on habits from Gua, like noises when she wanted something, and the way monkeys scratch themselves. No authoritative source for it though, it's what I've been told during a lecture back in college, and I think PlainlyDifficult mentions it too in their video about it.
Nice post! But to me your analogy does not really stand : bots are the ones catching up with human conversation in an "accelerated way", feeding on a corpus that predates them. Bots are not an invariant nature that netizens imitate.
I sincerely regret that I had only one upvote to give you. This shit is so insidious that IMO everyone should just simply stop doing it until they've thought it through a lot more.
> ...humans condition themselves in a far more accelerated way to behave like bots than bots are potentially able to do.
Than bots can condition themselves to behave like humans, I presume. They can already behave exactly like bots. :-)
My mind is blown. Thanks for sharing. Especially with the movie analogy. I’m a very movie person and I imitate my personality traits a lot based on characters on movies…
thanks for the awesome analogy, I always had the sinking feeling that the bots are finding it increasingly easy to fit in among the humans because the humans on social media act increasingly like bots.
That's definitely the future, personalized entertainment and social interactions will be big. I could watch a movie made for me, and discuss it with a bunch of chat bots. The future will be bubbly as hell, people will be decaying in their safe places as the hellscape rages on outside.
We're a long, long way from this. Stringing words/images together into a coherent sequence is arguably the easy bit of creating novels/films, and computers still lag a long way behind humans in this regard.
Structuring a narrative is a harder, subtler step. Our most advanced ML solutions are improving rapidly, but often struggle with coherence over a single paragraph; they're not going to be doing satisfying foreshadowing and emotional beats for a while.
You jest, but it really is the case. When your movie has a goddamn board of directors, you can be 100% sure it will be A/B tested until it transmutes the surrounding air into gold.
Maybe. But I think a lot of folks have a short term memory; it was not so long ago that Word2Vec and AlexNet were SOTA. Remember when the thought of a human besting a world-class player at Go was impossible? Me too.
We've come ludicrously far since then. That progress doesn't guarantee that innovation in the space will continue at its current pace, but it sure does feel like it's possible.
I actually wouldn't be surprised if the technology catches up to this faster than we realize. I think the actual barrier to large scale adoption of it will be financial and social incentives.
A big reason all the major studios are moving to big franchises is that the real money is in licensing the merch. The movies and TV shows are really just there to sell more merch. Maybe this will work when we all have high quality 3d printers at our desks and we can just print the merch they sell us.
The other big barrier is social. A lot of what people watch, they watch because it was recommended to them by friends or colleagues, and they want to talk about what other people are talking about. I'm sure that there will be many people who will get really into watching custom movies and discussing those movies with chatbots, but I bet most people will still want to socialize and discuss the movies they watch with other humans. FOMO is an underestimated driver of media consumption.
We’re probably 18 months away from this. We’re probably less than 5 years away from being able to do this on local hardware. AI/ML is advancing faster than most people realise.
We're probably a long way away from narrative, but dall-e for video is probably only a year or two away from now (they're probably training the model as we speak).
I get the feeling that creative sci fi used to kind of help inoculate us against these kinds of future but it seems like there's much less of it than there used to be.
"Black mirror" was good but it's not nearly enough.
The Machine Stops is eerily prescient - or perhaps just keenly observant of trends visible even at the time - but in fairness the humans in it are not socially isolated, as such; they do not converse with bots, but rather with each other. The primary social activity in the The Machine Stops is the Zoom meeting.
I do not look forward to the day when that story becomes an optimistic view of the future.
...what? 60 thousand dollars for a dedicated computer that you can't use is not everyone, not on their own computers, and is also a crazy large amount of money for nearly everyone. Sure there are some that could, but that's not what I said.
Most of the cost of a phone isn’t the processor, so probably closer to x1000. Hardware may get that much cheaper, but it was never guaranteed, and we’re not making progress as fast as we used to.
Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.
GPU TDP seems on the verge of going exponential, cost per transistor isn't really decreasing so much at the very latest nodes, and even that article seems to suggest it'd likely be decades before 300x flops/$
Plus, that's the energy costs involved when running a computer now worth 60k, I'm pretty sure that in the current socio-economic climate those power costs will surpass the initial acquisition cost (those 60k, that is) pretty easily.
I wanted to add that I was writing it metaphorically in a way, as in, seeing as how high those energy bills will be they might as well all add up to 60k.
Not sure about most of the people in here, but I would get really nervous at the thought of running something that eats up 3x300 watts per hour, for 24/7, just as part of a personal/hobby project. The incoming power bills would be too high, you have to be in the wage-percentile for which dropping 60k on a machine just to carry out some hobby project is ok, i.e. you’d have to be “high-ish” middle-class at least.
The recent increases in consumer power prices are a heavy blow for most of the middle-class around Europe (not sure about how things are in the States), so a project like this one is just a no-go for most of middle-class European programmers/computer people.
At full power 3 of those would cost me ~$3.50 per day ($0.15 per kWh is what I paid for last month's electricity, though I could pay less if I made some difference choices), I occasionally have a more expensive coffee order, or have a cocktail worth three times as much.
Things are getting more expensive here but nothing like the situation in Europe (essentially none of our energy was imported from Russia, historically ~10% of oil imports but that was mostly to refine and re-export, we have all the natural gas locally that we need) The US crossed the line into being a net hyrdocarbon energy exporter a while ago (unsure what the case is recently but it is at worst about at parity)
Eh, 60k is just a bit more expensive than your average car, and lots of people have cars, and that's just how things are today. I imagine capabilities will be skyrocketing and prices will fall drastically at the same time.
You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in.
Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work.
But for really transient loads you're better off just renting seconds-minutes on a VM.
There isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.
I believe you're confusing the amount of A100 graphics cards used to train the model (the cluster was actually made up of 800 A100s), and the amount you need to run the model :
> The model [...] is supposed to run on multiple GPUs with tensor parallelism.
> It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory.
I don't know what the price of a V100 is, but given $10k a piece for A100s we would be closer to the $60k estimate.
The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough.
Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at least 1.6kw PSUs etc etc.
You don't need a "decent amount" of PCIe-4 lanes. You just need 16 of them. And they can be PCIe 3.0 and will work just fine. Deep learning compute boxes predominantly use a PCIe switch. e.g. the ASUS 8000 box, which handles eight cards just fine. You only need a metric tonne of PCIe bandwidth if you are constantly shuttling data in and out of the GPU, e.g. in a game or exceedinyl large training sets of computer vision data. A little latency of a few hundred milliseconds moving data to your GPU in a training session that will take hours if not days to complete is neither here nor then. I suspect this model, with a little tweaking, will run just fine on an eight way RTX A5000 setup, or a five-way A6000 completely unhindered. That puts the price around $20,000 to $30,000. If I put two more A5000s in my machine, I suspect I could figure out how to get the model to load.
It also sounds like they haven't optimized their model, or done any split on it, but if they did, I suspect they could load it up and have it infer slower on fewer GPUs, by using main memory.
Which will work just fine with NVIDIA SWITCH and a decent GPU compute case from ASUS or IBM or even building your own out of an off-the-shelf PCIe switch and consumer motherboard.
And also, NVIDIA does not sell them to the consumer market whatsoever. Linus Tech Tips could only show one because someone in the audience sent theirs over for review.
You're grossly overestimating. People who make 60k annually are getting a bit rarer nowadays, it's not like everyone can afford it. For the majority of people it'd be a multi-decade project, for a few it might only take 7 years, very few people could buy it all at once.
Unpopular opinion: something will stop egalitarian power for the masses. I had high hopes for multicore computing in the late 90s and early 2000s but it got blocked every step of the way by everyone doubling down on DSP (glorified vertex buffer) approaches on video cards, leaving us with the contrived dichotomy we see today between CPU and GPU.
Whatever we think will happen will not happen. A less-inspired known-good state will take its place, creating another status quo. Which will funnel us into dystopian futures. I'm just going off my own observations and life experience of the last 20 years, and the way that people in leadership positions keep letting the rest of us down after they make it.
In what sense is the dichotomy between CPU and GPU contrived? Those are designed around fundamentally different use cases. For low power devices you can get CPU and GPU integrated into a single SOC.
That's a good question. I wish I could answer it succinctly.
For me, the issue is that use cases and power usage are secondary to the fundamental science of computation. So it's fine to have matrix-processing stuff like OpenGL and TensorFlow, but those should be built on general-purpose hardware or else we end up with the cookie cutter solutions we have today. Want to run a giant artificial life simulation with genetic algorithms? Sorry, you can't do that on a GPU. And it turns out that most of the next-gen stuff I'm interested in just can't be done on a GPU.
There was a lot of progress on transputers and clusters (the old Beowulf cluster jokes) in the 80s and 90s. But researchers came up against memory latency issues (Amdahl's law) and began to abandon those approaches after video cards like the 3dfx Voodoo arrived around 1997.
But there are countless other ways to implement concurrency and parallelism. If you think of all the techniques as a galaxy, then GPUs are way out at the very end of one spiral arm. We've been out on that arm for 25 years. And while video games have gotten faster (at enormous personal effort by millions of people), we've missed out on the low hanging fruit that's possible on the other arms.
For example, code can be auto-parallelized without intrinsics. It can be statically analyzed to detect contexts which don't affect others, and the instructions in those local contexts could be internally spread over many cores. Like what happens in shaders.
But IMHO the greatest travesty of the modern era is that those innovations happened (poorly) in GPUs instead of CPUs. We should be able to go to the system menu and get info on our computer and see something like 1024+ cores running at 3 GHz. We should be able to use languages like Clojure and Erlang and Go and MATLAB and even C++ that auto-parallelize to that many cores. So embarrassingly parallel stuff like affine rasterization and blitters would run in a few cycles with ordinary for-loops instead of needing loops that are unrolled by hand or whatever other tedium that distracts developers from getting real work done. Like, why do we need a completely different paradigm for shaders outside of our usual C/C++/C# workflow, where we can't access system APIs or even the memory in our main code directly? That's nonsense.
And I don't say that lightly. My words are imperfect, but I do have a computer engineering degree. I know what I'm talking about, down to a very low level. Wherever I look, I just see so much unnecessary effort where humans tailor themselves to match the whims of the hardware, which is an anti-pattern at least as bad as repeating yourself. Unfortunately, the more I talk about this, the more I come off as some kind of crackpot as the world keeps rushing headlong out on the GPU spiral arm without knowing there's no there there at the end of it.
My point is that for all the progress in AI and rendering and simulation, we could have had that 20 years ago for a tiny fraction of the effort with more inspired architecture choices. The complexity and gatekeeping we see today are artifacts of those unfortunate decisions.
I dream of a day when we can devote a paltry few billion transistors on a small $100 CPU to 1000+ cores. Instead we have stuff like the Cerebras CS-2 with a trillion transistors for many thousands of dollars, which is cool and everything, but is ultimately gatekeeping that will keep today's Anakin from building C-3PO.
Before any of the things you describe happen, most states will mandate the equivalent of a carry permit to be able to freely use compute for undeclared and/or unapproved purposes.
If by running models you mean just the inference phase, then even today you can run large family of ML models on commodity hardware (with some elbow grease, of course). The training phase is generally the one not easily replicated by non-corporations.
I know it's a sort of exaggerated paranoid thought. But like these things do all come down to scale and some areas of the world definitely could have the amount of compute available to make dall-e level quality full scale videos which we might be consuming right now. It really does make you start to wonder at what point we will rationally be able to have zero trust that not everything we watch online is fabricated.
Historically, hard-to-falsify documents are an anomaly, the norm was mostly socially conditional and enforced trust. Civilizations leaned and still lean on limited-trust technologies like personal connections, word of mouth, word on paper, signatures, seals, careful custody etc. I agree losing cheap trust can be a setback, just want to point out we’re adaptable.
I'm predicting that the upcoming Mac Pro will be very popular among ML developers, thanks to unified memory.
It should be able to fit the entire model in memory.
Combine that with the fact that PyTorch recently added support for Apple silicon GPUs.
Although memory capacity may matter more than speed for inference. As long as you're not training or fine tuning, the mac pro / studio may be just fine.
apart from the fact that you can't use any of the many nvidia-specific things; if you're dependent on cuda, nvcuvid, AMP or other things that's a hard no.
Comments like this make me feel like I'm losing my mind.
I think it's far more likely that in 10 years we'll all become more used to rolling blackouts, and fondly remember we all used to be able to afford to eat out, and laugh over a glass of cheap gin about how wild things were back in the old days before things got really bad.
10 years ago was a much more exciting and hopeful time than today. I remember watching Hinton show off what deep learning was just starting to do. It was frankly more interesting that high parameter language models. Startups were all working on some cool problems rather than just trying to screw over customers.
That's just technology. Economically, socially and ecologically things looks far brighter in 2012 than they do now, and in 2032 I suspect we'll feel the same about today, but far more dramatically.
We've already pass the peak of "things are getting better all the time!" but people are just in denial about this.
You're not alone - especially based on observations during the pandemic, it seem we are woefully unaware and unprepared for how fragile the structures supporting our current way of living are, and how easy they would collapse into a much worse state of living conditions when it comes to power and food, let alone luxuries and internet as we know it...
It also seems to me that most people would not be ready to give up more than 10% of their luxuries / way of living up-front in order to protect those structures and would continue to watch funny TikTok videos and post IG photos until the very moment their internet access goes out and doesn't come back.
I don’t know for you, but most of my online interactions are text based. Context of interpretation matter far much than the form of the content. If you know it’s easy to fake text exchanges, you might be more careful about text origin, and other contextual hints. Even it’s the syntax imitate your children verbal oddities, you may not necessarily run to comply thoughtlessly to an unusual demand you just receive by SMS from their phone number. Trust and check.
>> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers.
Do you mean train or run? My assumption was all these models could be run on most computers, probably with a simple docker container, as long as there is sufficient RAM to hold the network, which should be most laptops > 16gb ram.
Speaking of which, anyone have recommendations on pre-trained docker containers with weights included?
I'm not sure why you got downvoted. Yes, ASICs (either analog or digital) that have some model hardcoded in would probably make it feasible, but it won't be programmable which is the interesting part.
It's more likely, if not inevitable that these things will become ubiquitously available remotely, like Siri and Alexa. It's access that's important, not hosting.
Seeing those gigantic models it makes me sad that even the 4090 is supposed to stay at 24GB of RAM max. I really would like to be able to run/experiment on larger models at home.
It's also a power issue. The 4090 sounds like you're going to need a much, MUCH higher PSU than you currently use.. or it'll suddenly turn off as it uses 2-3x the power.
You'll need your own wiring to run your PC soon :-)
I think it is a stupid question, but does the power consumption needed by processors to infer compared to human brains demonstrate that there is something fundamentally wrong for the AI approach or is it more physics related?
I am not a physicist or biologist or anything like that so my intuition is probably completely wrong but it seems to me that for more basic inference operations (lets say add two numbers) power consumption from a processor and a brain is not that different. It’s like seeing how expensive it is for computers to infer for any NLP model, humans should be continuously eating carbs just to talk.
Around room temperature, an ideal silicon transistor has a 60 mV/decade subthreshold swing, which (roughly speaking) means that a 10-fold increase in current requires at least a 60 mV increase in gate potential. There are some techniques (e.g. tunneling) that can allow you to get a bit below this, but it's a fairly fundamental limitation of transistors' efficiency.
[It's been quite a while since I studied this stuff, so I can't recall whether 60 mV/decade is a constant for silicon specifically or all semiconductors.]
> but it seems to me that for more basic inference operations (lets say add two numbers) power consumption from a processor and a brain is not that different
Sure it is - it is too hard to figure it out based on 2 numbers number, but lets multiply that by a billion - how much energy does it take a computer to add two billion numbers? Far less than the energy it would take a human brain to add them.
Nvidia deliberately keeps their consumer/gamer cards limited in memory. If you have a use for more RAM, they want you to buy their workstation offerings like RTX A6000 which has 48G DDR6 RAM or A100 which has 80G.
What NVIDIA predominantly does on their consumer cards is limit the RAM sharing, not the RAM itself. The inability for each GPU to share RAM is the limiting factor. It is why I have RTX A5000 GPUs and not RTX 3090 GPUs.
It only gets expensive if you insist on sourcing it from enterprise vendors. The first 256GB I paid $2,400 for. The second 256GB I paid $1,200 a little over a year later. And the third 256GB I paid $800 about seven months later. I've got a workstation with 768GB DDR4 and I am considering upping that to 1.5TB if the prices on the 256GB sticks will come down.
What's the difference between Apple's unified memory and the shared memory pool Intel and AMD integrated GPUs have had for years?
In theory you could probably assign a powerful enough iGPU a few hundred gigabytes of memory already, but just like Apple Silicon the integrated GPU isn't exactly very powerful. The difference between the M1 iGPU and the AMD 5700G is less than 10% and a loaded out system should theoretically be tweakable to dedicate hundreds of gigabytes of VRAM to it.
It's just a waste of space. An RTX3090 is 6 to 7 times faster than even the M1, and the promised performance increase of about 35% for the M2 will means nothing when the 4090 will be released this year.
I think there are better solutions for this. Leveraging the high throughput of PCIe 5 and resizable BAR support might be used to quickly swap out banks of GPU memory, for example, at a performance decrease.
One big problem with this is that GPU manufacturers have incentive to not implement ways for consumers GPUs to compete with their datacenter products. If a 3080 with some memory tricks can approach an A800 well enough, Nvidia might let a lot of profit slip through their hands and they can't have that.
Maybe Apple's tensor chip will be able to provide a performance boost here, but it's stuck on working with macOS and the implementations all seem proprietary so I don't think cross platform researchers will really care about using it. You're restricted by Apple's memory limitations anyway, it's not like you can upgrade their hardware.
I downloaded the weights and made a .torrent file (also a magnet link, see raw README.md). Can somebody else who downloaded the files as well doublecheck the checksums?
For those of us without 200GB of GPU RAM available... How possible is it to do inference loading it from SSD?
Would you have to scan through all 200GB of data once per character generated? That doesn't actually sound too painful - 1 minute per character seems kinda okay.
And I guess you can easily do lots of data parallelism, so you can get 1 minute per character on lots of inputs and outputs at the same time.
These models are not character-based, but token-based. The problem with CPU inference is the need for random access to 250 GiB of parameters, meaning immense paging and orders of magnitude slower than normal CPU operation.
I wonder how bad it comes out with something like Optane?
It's not really random access. I bet the graph can be pipelined such that you can keep a "horizontal cross-section" of the graph in memory all the time, and you scan through the parameters from top to bottom in the graph.
Fair point, but you’ll still be bounded by disk read speed on an SSD. The access pattern itself matters less than the read cache being << the parameter set size.
You can read bits at that rate yes, but keep in mind that it’s 250 GiB /parameters/, and matrix-matrix multiplication is typically somewhere between quadratic and cubic in complexity. Then you get to wait for the page out of your intermediate result etc etc.
It’s difficult to estimate how slow it would be, but I’m guessing unusably slow.
That's pretty much what SLIDE [0] does. The driver was achieving performance parity with GPUs for CPU training, but presumably the same could apply to running inference on models too large to load into consumer GPU memory.
If you bother to set the permissions, I suggest to do it in a way that doesn't leave a time window during which it still is unprotected (note that non-priviledged processes just need to open the file during that window; they can keep reading even after your chmod has been run). Also, not sure what the point of `-U clear` was, that's setting the uuid for the swap, better leave it at the default random one?
Is there a reason why it is required to fill the swapfile with zeroes here? Normally you'd see something like "dd of=/swapfile bs=1G seek=3 count=0", creating a file of size 3G but with no space allocated (yet). It's much quicker to complete the setup this way.
I assume if you force the file system to allocate inodes you are likely to have a less fragmented file than if you create a sparse file that gets inodes assigned over time when each part is used.
On all the benchmarks of SSDs I've seen they perform 1.5 to 4 times better on sequential reads than on random reads. That's a much better ratio than HDDs, but still enough to care about it.
You're also likely to get less write amplification if your swap file is continuous.
Of course with all the layers of indirection it's a numbers game, you don't know if your file system allocates adjacent inodes, and you don't know how your SSD will remap the blocks. But all else being equal, trying to make the file as sequential as possible seems preferable.
But this does make me wonder if there's any way to allow a graphics card to use regular RAM in a fast way? AFAIK built-in GPU's inside CPU's can but those GPU's are not powerful enough
Unified memory exists, but it's not a magic bullet. If a page is accessed that doesn't reside on device memory (i.e. on the GPU), a memcpy is issued to fetch the page from main RAM. While the programming model is nicer, it doesn't fundamentally change the fact that you need to constantly swap data out to main RAM and while not as bad as loading it from the SSD or HDD, that's still quite slow.
Integrated GPUs that use a portion of system memory are an exception to this and do not require memcpys when using unified memory. However, I'm not aware of any powerful iGPUs from Nvidia these days.
Sure. Makes sense. So I guess for discrete GPUs the unified memory stuff provides a universal address space but merely abstracts the copying/streaming of the data.
There does seem to be a zero copy concept as well and I've certainly used direct memory access over pcie before on other proprietary devices.
It's just crazy how much it costs to train such models. As I undestand 800 A100 cards would cost about 25.000.000 without considering the energy costs for 61 days of training.
Still a bit to expensive for my sideproject ; ) To be honest it seems only big corp can do that kind of stuff. By the way if try to do hyper parameter tuning or some exploration in the architecture it becomes guess 10x or 100x more expensive.
AWS has them in US-EAST1 for $9.83/hr spot with 96 CPU cores, 1152GB of ram, 8 A100s with 320 GB of RAM, 8TB of NVME, and 19 Gbps of EBS bandwidth to load your data quickly.
Side note: Yandex search is awesome, and I really hope they stay alive forever. It's the only functional image search nowadays, after our Google overlords neutered their own product out of fear over lawyers/regulation and a disdain for power users.
You can't even search for images "before:date" in Google anymore.
Yandex Image Search is today is what Google Image Search should have been.
End of the day I’ll use what actually gets the job done.
Same goes for OpenAI and Google AI. If you don’t actually ever release and let others use your stuff and end paralyzed in fear at what your models may do then someone else is gonna release the same tech, and at this rate it seems like that’ll be Chinese or Russian companies who don’t share your sensibilities at all, and their models will be the ones that end up productized.
IMO the main reason these companies don't release their models is not ethical concerns but money:
- NVIDIA sells GPUs and interconnect needed for training large models. Releasing a pretrained LM would hurt sales, while only publishing a teaser paper boosts them.
- Google, Microsoft, and Amazon offer ML-as-a-service and TPU/GPU hardware as a part of their cloud computing platforms. Russian and Chinese companies also have their clouds, but they have low global market share and aren't cost-efficient, so nobody would use them to train large LMs anyway.
- OpenAI are selling their models as an API with a huge markup over inference costs; they are also largely sponsored by the aforementioned companies, further aligning their interests with them.
Companies that release large models are simply those who have nothing to lose by doing so. Unfortunately, you need a lot of idle hardware to train them, and companies that have it tend to also launch a public cloud with it, so there is a perpetual conflict of interests here.
Their "moral" reasoning behind not publishing models is simply laughtable because they do sell API access to them to anyone who can pay. And "bad guys" generally have money.
They can (and do) revoke API access from bad guys. They can't do that to downloaded models. Look, I don't like what OpenAI does, but "API access, but no model download" makes sense if you are worried about misuses.
Every company out there says it will "revoke API access for misuse", but do they have transparency reports? Who do they even consider bad guys and what do they consider as misuse?
I would be totally on their side if their reasoning was that they dont publish models to compete with FAANG more efficiently and get more income for their research, but this moral reasoning just sounds completely fake because bad actors do have funding to train their own models.
Examples of "real cases of misuse encountered in the wild" include "spam promotions for dubious medical products and roleplaying of racist fantasies".
Yes, some bad actors can train their own models, but OpenAI can't do much about that either way. It is doubtful whether spam promoters of dubious medical products can, at least for a while.
It would be better for misuse to be criminalized and taken care of by national governments, rather than leave it to for-profit companies to decide what is or isn't "misuse".
Personally, I think using AI to manufacture advertisements on demand is misuse... but will Google agree with me?
Bad actors still can get access to such models.
It even makes them more dangerous than it would if everyone had access to them.
Here's an alternative: progressively release better and better models (like 3B params, 10B, 50B, 100B) and let people figure out the best way to fight against bad actors using them.
He was an author of fiction. They usually write a lot of stuff that is on some deeper level "true", even though it is on the surface fictional... And a lot of stuff that isn't.
Or sometimes both, because it's only part of the truth. Maybe the complete version should be: "An armed society is a 'polite' society, but with very frequent killings and not-infrequent massacres." I think I prefer living in a less "polite" society.
when you release a project into the wild under a permissive license, aren't you essentially washing yourself from any "liability" ?
> MIT
" IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE."
don't commercial licenses have same/similar wording so what liability are you talking about?
So they do that because they're doing it for free. Otherwise they couldn't be generous with their work--those licenses are about permitting the generous intent.
The “ethical concerns” thing is just a progressive-sounding excuse for why they’re not going to give their models away for free. I guarantee you those models are going to be integrated into various Google products in some form or another.
> Google overlords neutered their own product out of fear over lawyers/regulation
What kind of lawyers/regulation do you have in mind? If anything, I'd find the opposite: lawyers and copyright holders should be grateful for such a tool that - when it was still working - allowed you to trace websites using your images illegally.
Now they all use Yandex for this purpose, with relatively good results.
Oh I see. What I'm looking for is the reason why they broke the reverse image search. It was working well many years ago but some time after that they switched it to some strange image classifier (I upload an image of an apple to find exactly the same image to track its license of origin, and it says "possibly an image of an apple" - oh thank you Google I didn't know that.)
> Tineye works reasonably well, for finding exactly the same image (including different resolutions, crops, etc.)
Tineye is definitely better than Google with crops, etc. Google reverse image search seems to have more data, but it seems much less able to recognize even basic modifications to the input.
They used to have in their AI ethics department some of the most anti-AI progressives. They picked on everything - biased training data, discriminatory usage, consuming too much energy to train, models are just stochastic parrots, etc. while forgetting to mention any effort to mitigate the problems (of course these are real concerns and being under intense research) Now these critics are fired, but Google must have learned to fear them.
If they let everyone use the latest models, critics could uncover ugly biases in 10 minutes. Then Google would have to do damage control. These models are very suggestible. You can induce them to make fools of themselves.
IIRC it was mostly from groups like Getty images. They and other image licensing companies didn't want google showing their images in search results. They claimed it was copyright infringement and given the absolute state of IP law in the US they could have made Google's life very difficult.
We're talking about reverse search, right? (Because "normal" image search still kind of works, it's reverse search that is completely broken.) In this case, you already have the copyrighted image, and if you find out that the same image is on Getty Images, then all the better as you can check it license. Also, it's better for GI as it gives them more exposure, and the kind of companies who use GI are very unlikely to pirate images.
You look at OpenAI and how they don't release their models mainly because they fear "bad people" will use them for "bad stuff." This is the trend in the west. Technology is too powerful, we must control it! Russia is like... Hey, we are the bad guys you're talking about so who are we keeping this technology from? The west has bigger language models than we do, so who cares. Also their attitude to copyright and patents, etc. They don't care because that's not how their economy makes money. Cory Doctorow's end of general purpose computing[1] and locked down everything is very fast approaching. I'm glad the Russians are around and aren't very interested in that project.
[1]https://csclub.uwaterloo.ca/resources/tech-talks/cory-doctor...