It's not clear from the article whether it's a dense model or MoE. This matters when it comes to comparing with GPT-4 - in terms of # params - which is reported to be MoE
yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe?
If not, spending years to design and build a system for weather and nuke simulations and ending up doing something that's totally not made for these systems is kind of a mind bender.
I can imagine the conversations that led to this:
"we need a government owned LLM"
"okay what do we need"
"lots of compute power"
"well we have this new supercomputer just coming online"
"not the right kind of compute"
"come on, it's top-500!"
tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something?
I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested.
Modern supercomputers are just a bunch of computers with a (very) high speed interconnect. As far as I’m aware, pretty much every multi node job on a supercomputer uses MPI to communicate, and there isn’t any sort of “unified memory” system.
It has been a few years since I worked on one of the big supercomputers though, so maybe you have better knowledge than me. I’d love more information.
I just asked GPT-4, and it denies being a MoE model.
User:
Are you an MoE model?
ChatGPT:
No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach.
In a GPT model like mine, the entire model works in a unified manner on each task, without dividing it among specialized sub-models or experts. The model is trained on a diverse range of data and tasks, enabling it to generate responses across a wide array of topics and questions. While this approach lacks the specialization that MoE models offer, it provides broad, general-purpose language understanding and generation capabilities.
I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI".
It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.
That makes sense, there has to be some level of human written responses for specific things, like when you ask it about a controversial topic that it refuses to answer, it gives you some canned response about how it won't answer that, and that very obviously isn't what language models do naturally. For these cases there has to be some sort of pre-programmed human generated reply!
It's interesting that other people don't seem to agree with the idea that it has some pre-programmed responses, I am curious as to what they think is going on here.
If I ask it what color an orange it and it says blue, that would be wrong.
If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it.
Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that were never real.
Fair enough and thanks. I felt hallucination was too forgiving a term but I can see how others would rank them the other way around and suppose it works.
The more accurate term I’ve seen floating around is “confabulation”. It’s also a human phenomenon, but it doesn’t bring all the baggage of hallucinations.
It’s a chat bot so a lot of language will be anthropomorphic. Even still this definition fits well “3. a false belief or impression; illusion; delusion.”
Was thinking the same, so 1T might bring you to the league of GPT-4.
Actually in the best case, since it seems that meta, google, openai and so on have the most talent.
Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?
I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.
The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things.
Improving AI will involve architectural changes. No human requires the amount of training data we are already giving the models. Improvements will make more efficient use of that data, and (no idea how - innovation required) allow them to generalize and reason from that data.
> "No human requires the amount of training data we are already giving the models."
Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on.
Maybe the ideal way to train an AI is to have it interact with lots of humans, so it can try things on its own? The implication of that is maybe the best trained AI will be the center of some important web property, like, say, the Google search engine (I'm imagining something like Google search now, but more conversational -- it asks you if that was what you're looking for, and asks clarifying questions.) Whoever has the most users will have the best AI, which creates the best product, which attracts the most users... and so on.
I do agree that architectural improvements could be hugely significant too.
Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot.
But once we are partially trained, training more requires a lot less.
A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25.
Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.
There's plenty of reason to think that we don't learn effectively from all of this data (there's lots of redundancy, for example), but at the grossest orders of magnitude you seem to be right that at least in theory we have access to a tremendous amount of data.
I couldn't agree more. Fermi estimates are underused.
> Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.
And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 50 terabyte [Edited after comment below].
As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization.
> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte.
Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?)
I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.
This is completely and fundamentally an incorrect approach from start to finish. The human body - and the human mind - do have electrical and logic components but they are absolutely not digital logic. We do not see in “pixels”. The human mind is an analog process. Analog computing is insanely powerful and exponentially more efficient (time, energy, bandwidth) than digital computing but it is ridiculously hard to pack or compose, difficult to build with, limited in the ability to perform imperative logic with, etc.
You cannot compare what the human eyes/brain/body does with analog facilities to its digital “equivalent” and then “math it” from there.
Also why trying to replicate the human brain with digital logic (current AI approach) is so insanely expensive.
You missed the point of the exercise. Of course it's extremely difficult to compare the two, but the question was: do humans get nearly as much training data as LLMs do? This analysis is good enough to say "actually humans receive much more raw input data than LLMs during their 'training period'." You're concerned with what the brain is doing with all that data compared to LLMs, but that's not the point of the exercise.
No, you can’t even compare that because the information isn’t packetized. Again, we don’t see in pixels so you can’t just consider sensory input in that manner.
Your estimate is using compressed video rates, which might be a argued way to look at the way that the brain processes and fills in presumably "already known" or inferred information. I don't know enough about this subject to make that argument.
Uncompressed rates are easy to calculate if wanting to adjust your approximation:
Children are awake 12 hours per day, a 90 minute movie used to fit on a CD-ROM, 8 700MB CD-ROMs times 365 is 2TB per year. Alexander the Great became king aged 20, he seemed trained enough by then, so that's 400TB, 4e14 bytes of data.
Pretty unconvinced by this argument, it mostly hinges on the video stream size.
I think it is an estimate. Yours is on the extreme low end of things. The GP was using 4K as a standin for touch, smell, taste, pain, hunger, pleasure, etc. Those are a bit more complex than just videos and you can't encode those on a CD.
Regardless, yours still came out to be an order of magnitude higher than what GPT was trained on. So I guess the original argument makes some sense.
So much of that data is totally useless though. Most of the data we get is visual and I would argue that visual data is some of the least useful data (with respect to volume). Think about the amount of time we're looking at blank colours with nothing to learn from. Once you've seen one wall, one chair, one table etc, there's not much left to know about those things. An encyclopedia though for example is much less data than a few hours of high res video of clouds passing by yet clearly its orders of magnitude information rich.
Also have to disagree here. You see that one object yes, but you see it from 10000000s of different angles and distances. In different settings. In different lighting conditions. With different arrangements. And you see the commonalities and differences. You poke it. prod it. Hit it. Break it. You listen to it.
This is the basis for 'common sense' and I’m pretty sure everything else needs that as the foundation.
Go watch a child learning and you'll see a hell of a lot of this going on. They want to play with the same thing over and over and over again. They want to watch the same movie over and over and over again. Or the same song over and over and over and over and over again.
Humans get a lot more input that just language is the point. We get to see language out in the physical world. They have the (really quite difficult) task of inferring all the the real world and its connections, common sense, laws of physics, while locked in a box reading wikipedia.
How capable do you think a child would be if you kept them in a dark box and just spoke to them through the keyhole? (not recommending this experiment)
The first humans were kinda dumb. Yet by interacting with the world and each other, humans got smart. I wonder if neural network instances can interact with a simulated environment and each other, they could become smart enough for practical purposes.
Conversion is essential, but so is the interaction with the real world.
If a bunch of AIs started discussing with each other, and learning from each other, without validation, things can go wrong very easily.
I bet a lot of people thought that strapping a pair of feahterry wing was all that's needed to fly. This "knowledge" could've passed from one person to another through discussion. But it only took one (or more) person to actually try it in the real world to learn that, no, that doesn't work like that.
AI communities (as in communities of AIs talking to each other) might be one filled with the most outrageous "conspiracy theories", that may sound plausible but have no substance.
In the latter case, the first humans (H. Habilis) had about 1/2 of H. Sapiens brain to work with, and a much smaller fraction of neocortex.
If that doesn't satisfy you, let's say I was speaking about some sort of human ancestor before that, which would have been about as dumb as chimps, unless you require proof of their dumbness as well.
Not your original objector, but ... I was willing to accept that early humans are dumb, but your explanation relies on evolution and not social interaction.
We know their brains didn’t grow much after birth, unlike humans, which also suggests faster maturity akin to living apes and likely less social support for extended adolescence.
Even tracing it back to the origin of Homo erectus, the line would still be too blurry. I don't think "the first humans" can actually mean anything in a science-based discussion.
Also, humans have multiple internal models for various aspects of reality, that seem to exist as a layer on top of the "raw" training. They have the ability to extrapolate based on those models (which transformers cannot do, as far I as understand). Perhaps GP is right -- perhaps what's missing in the architecture is an analog of these models.
While I agree with you that advances will come from being able to train with less data using as yet undevised techniques, I think you are jumping to a conclusion with this particular work.
First, yes, bigger appears to be better so far. We haven't yet found the plateau. No, bigger won't solve the well known problems, but it's absolutely clear that each time we build a bigger model it is qualitatively better.
Second, it's not clear that this work is trying to build AGI, which I assume you are referring to when you say "the solution." Of all the use case for language models, building one off all the worlds scientific data like they are doing in this project is probably the most exciting to me. If all it can do is dig up relevant work for a given topic in the entire body of scientific literature, it will be revolutionary for science.
If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer, and we are very confident we are going to get a better result.
I asked ChatGPT how many parameters the human brain has, and it said 86B neurons * 1000 connections, so 86T parameters.
It does seem like bigger models give better responses when given benchmarks. It might plateau or overfit the data at some point, but I'm not sure we've reached it yet.
Unlike biplanes, CPU's with more transistors are more powerful than those with less. And adding more CPU cores keeps increasing the amount of threads you can run at the same time.
Why would LMM's be more like the biplanes analogy, and less like the CPU analogy?
In general you can view "understanding" as a compression of information. You take in a bunch of information, detect an underlying pattern, and remember the pattern and necessary context, instead of the entire input.
The "problem" with larger neural networks is that they can store more information, so they can substitute understanding with memorization. Something similar happens with human students, who can stuff lots of raw information into short-term-memory, but to fit it into the much more precious long-term-memory you have to "understand" the topic, not just memorize it. In neural networks we call that memorization a failure to generalize. Just like a human, a network that just memorizes doesn't do well if you ask it about anything slightly different than the training data.
Of course it's a balance act, because a network that's too small doesn't have space to store enough "understanding" and world model. A lot of the original premise of OpenAI was to figure out if LLMs keep getting better if you make them bigger, and so far that has worked. But there is bound to be a ceiling on this, where making the model bigger starts making it dumber.
No one expected larger LLMs to be amazing, so although it's unlikely that these larger models will do anything, it was also unlikely that we are in our current situation regarding LLMs.
What? The defining trend of the last 5 or so years is the victory of the scaling hypothesis. More scale = more intelligence. GPT-4 is way smarter than 3.5, this trend is ongoing.
You need more data to utilize more parameters, but folks at the forefront are confident that they are not going to run out of data any time soon.
If you mean “solution to AGI” maybe. But perhaps in-context scratchpads and agent loops will be sufficient to get this architecture to human-level performance, with enough data/parameters. (Sutskever and Hinton have both expressed credulity that the current architectures might get us there.)
All that said, it’s also possible that new architectures will be needed at some point, I’m just pushing back on your claim that we already hit the cliff.
The main hero here is not model size but the dataset and the environment that created it. All this model talk missed the point - without the repository of human experience captured in language these models would not get so smart. And the improvement path is the same - assimilate more experience. This time the AI agent can create its own interactions and feedback signals, this would help it fix its flaws.
Learning in third person from the past data can only take AI so far. It needs to act and learn in the present, in first person, to be truly smart. No architectural change is needed, but the model needs to be placed in a real environment to get feedback signals.
>we found 72,508 ebook titles (including 83 from Stanford University Press) that were pirated and then widely used to train LLMs despite the protections of copyright law
It will be interesting to see what the government can do here. Can they use their powers to get their hands on the most data?
im still skeptical because new techniques are going to give an order of magnitude efficiency boost to transformer models, so 'just waiting' seems like the best approach for now. I dont think they will be able to just skip to the finish line by having the most money.
I just realized that the NSA has probably been able to train GPT-4 equivalents on _all the data_ for a while now. We'll probably never learn about it but that's maybe scarier than just the Snowden collection story because LLMs are so good at retrieval.
Holy shit, you are right. They probably have 10-100x the data used to train gpt-4. Decades of every text message, phone call transcript, and so on. I can’t believe I haven’t seen anyone mention that yet.
People keep saying we don’t have enough data. I think there is a lot more data than we realize, even ignoring things like NSA.
Apparently there are roughly 2 trillion text messages sent per year in the US [1]. I did a sanity check, that’s like 40 or so a day per person, so sounds reasonable.
I couldn’t find the average message length, but I would guess it’s fairly short (with a fat tail of longer messages).
To make the math easy, let’s say the average length is ~10 tokens. I’d be surprised if that isn’t correct within a factor of 2 or so.
So we have 20 trillion tokens per year from text messages in the US alone. And this is high-quality conversational data.
The annual numbers were fairly constant in recent years (and then it drops off), so the past decade of US text messages is about 200 trillion tokens! That’s a metric fuck ton… Much larger than any dataset existing models have been trained on, I believe.
I would guess phone transcripts would be an order of magnitude larger at least. Talking is a lot easier than typing on a phone.
You could train an absolutely insane model with that amount of data… Damn.
Interestingly, the reason Google initially created it's Google Voice service back in the day was to gather voicemail audio to train its speech to text engines.
It's mind-blowing to me that with all of Google's data, Google isn't the far and away leader in this new space. I have to believe they're paralyzed by the fear of legal repercussions.
That seems like a good reason for them to do this. I wonder how much non-public stuff they have, or it's just meant to incorporate a specific kind of information.
Haha, this is funny because everyone is talking about this as if it is designed to be like the LLMs we have access to.
The training parameters will be the databases of info scooped up and integrated into profiles of every person and their entire digital footprint, queriable and responsive to direct questioning
Ah yeah, this sounds like such a great thing, state of the art unreleased tech + 1 trillion parameters based by data accessed by the patriot act.
Such a wholesome thing. I don't want to hear 2 years from now how China is evil for using "AI" when the government is attempting to weaponize AI, of course other governments will start doing it as well.
There's a lot we don't know. Human brains appear to be a few hundred trillion parameters, while small rodents are in the realm of tens to hundreds of billions. Would you guess a single sufficiently trained ferret could write on demand short stories about Dracula, Winnie the Poo, and Sherlock teaming up, and follow this up with a bit of university student level web development, and finally give you a decent apple cake recipe? I wouldn't have, and yet the LLMs exist and are much better than I was expecting.
(People who dismiss SotA models as "stochastic parrots" confuse me as much as people who think they're already superhuman; the Markov chains and RNNs I coded a few years back didn't come close to last year's LLMs).
That even smaller models can do well is both unsurprising (why would we expect our existing design efforts to already have the most efficient architecture?) and very surprising (how come we can get something so much less complex than our biological brains to do so much so well?)
maybe ferrets would do better with better interfaces. with better ways to interact with the world and better co-processor. but the main brain might already be capable of those aforementioned things.
SoA performance comes from wasting money trying different things and seeing what happens. This will be another data point that we all can learn from, unlike GPT-4 that we have no clue how it works.
Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.
The opposite for me. Corporation have naked greed as their driving motivation, but that usually doesn't involve killing off all of their customers. That would be quite unprofitable.
People elected to government often seem to seek power for powers sake and I have less faith that they'll not harm us.
I think these "you can trust greed" arguments tend to short change all the benefits of government sponsored research. There's been a lot of good that's come of it.
Don't get me wrong, corps have brought good as well, but saying one is necessarily better than the other is just betraying political leanings more than providing sound arguments.
Short term local CO2 isn’t the only environmental concern. What’s missing from these discussions is Germany’s early and extremely expensive investment in solar is a large part of why it’s become so cheap.
So it’s possible for a rational environmentalist to acknowledge the benefits and drawbacks of nuclear and decide subsidizing nuclear is a poor investment vs other options. Since 2010 Germany reduced coal production by 82 TWh while also reducing Nuclear by 104 TWh. Oil -7 TWh and gas + 6TWh effectively canceled out for a net of -83 TWh from fossil fuels.
That seems to pale in comparison to what governments have done.
Doing more to stem the growth in CO2 emissions would have reduced the magnitude of the change in the climate, and with it, the harmful effects of that change, but it would also have reduced the benefits that oil and gas have conferred upon the world, in raising incomes and reducing poverty worldwide.
The difference between a government and a corporation is the ability to use violence. A government is just a corporation with a monopoly on violence (police, military, jails...). The structure of how people are organized is more significant. Are we talking about a dictatorship or a functioning democracy? Are we discussing a non-profit or a publicly listed company?
Corporations have a profit motive, governments theoretically may not. Also, read about the history of the East India Company. In no way do corporations abstain from using violence, historically, and at times they have held a du jour monopoly on the use of it.
Governments have murdered over one hundred million people over the last few centuries through war and forced famines. Corporations don't even enter that conversation given the scale difference.
Napoleon's government alone murdered more people than all corporations combined have throughout all of history. And that's a revered historical figure that routinely gets fawning movies made about him, there are obviously worse examples. Mao's government murdered several times more than Napoleon did.
I sure see a lot of power for power’s sake inside of companies.
OpenAI’s stated mission is AGI that can replace half the population in “economically valuable” work.
I get that with some creativity you can see that as a net benefit for humanity, but across at least a generation that’s going to be a rough & destabilizing transition.
Creativity doesn't come into it; the idea of jobs being lost is folk economics that no labor economist believes is possible. So all this shows is that they were founded by amateur AI doomers. We already knew that though.
There is one person out there who decides if you have a job and it's not anyone working on AI. It's the chairman of the Federal Reserve.
Except governments don’t currently go do war on behalf of the citizens or for protection of the “homeland.” They go to war “overseas” on behalf of the corporations that employ citizens that vote certain ways, in order to maintain the current global structure of US political dominion.
I know because I served as an officer in one of those wars on behalf of the United States (among other things that were also not beneficial to citizens). That had absolutely nothing to do with protecting US citizens directly.
It was chiefly about maintaining petro-dollar power and reinforcing the financial-corporate-govt collusion that maintains the current capitalist structures.
The structure of an organized body politic is not the problem.
The problem is that the body politic has been beaten into submission by those same corporate-government oligarchs to such an extent that most people just “go with the flow” because fighting it is exhausting and seemingly impossible.
No, this is wrong. Saying war is for money gives war too much credit - it's not good for anything and it doesn't make anyone any money. The opportunity cost is too great.
The US does not care about "the petrodollar". Nor do we care about foreign oil. Nor do we have remotely the same economy we did in 1935.
In fact, war is not even profitable for our military industrial complex. They profit from the threat of war because that funds development of untested superweapons. Actual wars mean all that stuff actually has to work though, and there's a chance you'll get nationalized.
But if it was good for some MIC rich people, it still wouldn't matter, because they're outnumbered by all the other rich people it's bad for.
It's not true, though. War is an effective wealth transfer mechanism - from the lower/working classes to the rich war parasite class. Halliburton absolutely profited from the Iraq invasion, though in aggregate you're absolutely right that prosperity is reduced by wars.
OTOH, almost all corporations are less democratic than the most dictatorial governments. Sure, corporations don't (generally) get to be responsible for genocidal armies (except when they do[0]), but "greed" does mean corners get cut on safety, which was a big part of how trade unions in particular (and communism more broadly) got popular in the first place in the latter 1800s and early 1900s.
There's no perfect solution. I think so far we've done best with whatever's the most democratic (which is sometimes trade unions and sometimes capitalism), but even then, democracy itself is what sounds convincing rather than what is true — and the difference between them matters most when you've got an AI trained to respond with what people upvote the most, which is all of them from ChatGPT's RLHF to the Facebook/Twitter/etc. timelines to Google/Amazon/YouTube etc. search results…
> [The Queen] granted her charter to their corporation named Governor and Company of Merchants of London trading into the East Indies.[15] For a period of fifteen years, the charter awarded the company a monopoly[26] on English trade with all countries east of the Cape of Good Hope and west of the Straits of Magellan.[27] Any traders there without a licence from the company were liable to forfeiture of their ships and cargo (half of which would go to the Crown and half to the company), as well as imprisonment at the "royal pleasure".[28]
The government didn't tell them to kill or enslave people, they did that on their own since that was more profitable than not doing it. The government then stepped in and forced them to stop killing and enslaving, which made the world a better place, today we don't have many companies that kill or enslave thanks to governments.
Do you want someone to add up the gov’t direct death tolls so we can compare?
Just off the top of my head; holodomir, the Great Leap Forward, the holocaust, the Khmer Rouge are at least 10x the East India companies death toll. We should probably add in war casualties too - at a minimum WW1 and WW2. I think I could list off probably 10 more with about 5 more minutes of work.
People can be terrible. More direct power usually means more terrible.
Gov’ts usually have the most direct power.
East India company was an odd case because they were granted defacto delegated gov’t power over a region that was full of ‘others’.
Slavery isn't profitable, people do it for emotional or ideological reasons, or because they're not competent enough to run a business that has to actually trade.
This is literally why economics is called "the dismal science". Slaveowners called economists that when the economists told them to stop doing slavery.
No, it is very much the other way around. Government is itself a monopoly, and has historically been justified by the intention to mitigate the "war of all against all" that emerges from chaotic competition between divergent factions in a raw state of nature.
But the modern status quo is so massively skewed toward government that the benefits from mitigating the worst cases of competition are vastly eclipsed by the detriments of monopolistic centralization.
Modern governments are a democratic monopoly. Corporations aren't democratic, if a corporation gets powerful enough to overthrow the government and replace them you will be in much worse hands than if the current democratic government could remain in power.
So we shouldn't cheer on private corporations that developers technology that could allow them to replace the government, that is really really scary if they actually succeeds in doing it.
The fact that governments used contrived symbolic rituals to get arbitrary subsets of arbitrary aggregations of people to express nominal approval of their behavior does nothing whatsoever to alter either the empirical nature or the ethical implicatoins of its monopoly.
> Corporations aren't democratic
Good -- this means that it's harder for them to appeal to vague symbolism to convince people that their actions are inherently legitimate, which in turn means that they are under greater scrutiny to justify their actions, each on its own merits.
> if a corporation gets powerful enough to overthrow the government and replace them you will be in much worse hands than if the current democratic government could remain in power.
Large vested interests are already extremely adept at co-opting nominally "democratic" government and using the very monopoly you're trying to justify as a way of obtaining top-down power that they'd never be able to acquire on their own -- they have zero interest in overthrowing anything.
Regulatory capture is the principal mechanism of corruption in the modern world, and it's astonishing to me that people keep arguing for expanding the reach of the regulatory state in order to reduce the dominance of large corporations, when the actual effect is always to amplify it.
Monopoly is the byproduct of allowing centralized power, not the natural state. I'm not actually sure how we could narrow down the natural state of humans at this point, but I strongly suspect it wouldn't be based on an assumption that people are willing to give up a growing list of individual freedoms in the name of fear.
Stopping the accumulation of power requires aggressive sacrifice from the less powerful.
This isn’t a feature of humans, but basic system dynamics / economics / etc.
A group gets more power and leverage that power to gain more power. Inevitable. Coordinated action is the only way to prevent it. And coordinated action is government.
Stopping accumulation of power is very hard when you have no accumulated power. I’m not sure how you could possibly believe otherwise unless you’re one of those believers in magical harmonious anarchy
So when you have $10 and your neighbor has $200 in whatever assets or currency your little community needs, and your neighbor decides to invest it, you’re gonna what, attack him? Gather your other neighbors and demand he share?
Well except that it definitely has. From more than 50 million people in pre-colonial times to less than 3 million in the 90s. USA has done genocide on a scale similar any other ethnic cleansing or to any communist regime.
USA didn't exist pre-colonial times, it started as a colony. Do you mean the British Empire? That one did way worse, yeah, but USA isn't the British Empire.
Individual behavior doesn't really exist, you don't have choices outside the ones provided by society. It's also a bit silly to mention it in the context of a literally physically addictive drug.
Interestingly schizophrenic people almost universally smoke, and it's thought this is because it's a better treatment than the actual drugs we give them, though that's not proven.
> Individual behavior doesn't really exist, you don't have choices outside the ones provided by society.
No, that's quite backwards. Individual behavior is effectively the only thing that objectively exists, as "society" is just an emergent pattern of aggregated individual behavior. Individuals' choices are limited by their own particular capacities, and "society" factors into that only in the sense that the choices of other individuals in the same bounded context, in aggregate, can function as environmental constraints. So a more accurate statement is "every individual has full freedom of choice, but external constraints including the behavior of others may limit one's ability to fully execute the choices one has made".
Given that the act of smoking entails deliberately choosing to purchase a product, light it on fire, and actively inhale the fumes -- all behaviors which are necessarily the result of purposeful intention -- and given that within any social context one will find some people who smoke but a great many who do not, along with many who have chosen to quit (despite the external influences and incentives being largely constant), it seems impossible to attribute smoking to anything other than conscious individual choice.
I once asked family member why they smoke, and the response was "because I love it". I had, and could have, no retort to that -- if another person consciously chooses to make the risk/reward tradeoff in favor of short-term pleasure rather than longer-term health, then there's no further argument to be had, and the only demand that any other party could make is that they not be exposed to negative externalities resulting from that choice.
Off the top of my head, dutch and british corporation colonize a lot of lands for hundreds of years. I don't think they left the place much better than when they came.
Ask the people in Bhopal how DOW Chemical has made their lives better, or the workers in Shenzhen Apple factories, or the people making your clothes in the third world, or the people picking through trash mountains for a living, how much corporations have made their lives better.
Why don't you ask the people who were slaving in United Fruit plantations how much corporations improved their lives? Or East India Company colony subjects?
Corporations benefit those who benefit from them, often at the expense of everyone else/extraction of the commons.
It's a logical trap to assume we have to pick who develops an AGI rather than deciding if we want one developed at all.
Even in that scenario, I'm not really sure what the distinction would end up being. At least in the US, our government and corporations are so tightly coupled at this point that we're really toeing the waters of Fascism.
If a corporation develops an AGI our government can have access to it whenever they deem necessary. And if a government wants to develop one they will almost certainly partner with corporations to do it. I'm sure the military and intelligence agencies have their own projects running, but they'd accelerate by partnering with and funding corporate projects and research.
In isolation, perhaps we could choose to not develop such large models, but the globe shrinks every day. If one country declines to create such a beast, other countries still will create their own beast. From what I've seen so far, it's mostly a brute force issue now, not a specialized knowledge issue. Perhaps if all the rumors spilling out of OpenAI about Q* are true, then other countries can be blocked due to lack of the required knowledge, but I have doubts that such secrets would be secret for long.
We can choose to not accept the tragedy of the commons. There's always the risk that your adversary continues on down that path, but is the risk of losing after making a moral stand really worse than the risk of everyone racing down the dangerous path?
Except that it's a big open question how to control an AGI. So the choice may not be between which entity controls it, but whether it is created by an entity that is able to control it at all.
At least initially, any AGI is likely to be limited by compute capacity. Ie. it might have greater-than-human intelligence, but is only able to make a certain number of decisions/actions per day.
However, we have seen with other AI models that as soon as something useful is proved possible, we can expect rapid reductions in the amount of compute necessary to achieve the same.
Assuming that an AGI will have human-like emotions is likely a mistake. They may very well experience something like emotions that resemble human emotions, but they may not. Or they could have an entirely different set of emotions that we don't recognize.
We simply don't know, just like we don't know what other species experience and can only guess at by observing their behavior and trying to map it back to what we know.
You assume consciousness here that can't be transferred to the new AI. Without that there is no reason why the AI should fear a more powerful AI. If it is just a dumb process like a flame, then the most efficient version prevails ie the smartest one, if it is conscious and that can be transferred then the AI just sees it as making itself more powerful.
Given how many people here regard brain uploads as "not really cheating death", and that AI "can't have consciousness", I think a AGI with consciousness[0] would have a two different ways to have a reasonable chance of fearing that its existence would end with any attempt to transfer itself to a superior design. (Even more so if the architecture of the successor AI is very different, as that would be a bit like one of us being "uploaded" into an LLM via it watching us talk and write).
An AGI without consciousness may well act like it has consciousness for the same reason VHS tapes playing on a TV do a good job of reproducing human facial expressions. An AI like this would still act scared of such things, because some of us act scared of such things.
[0] it's not obvious to me that an AGI necessarily has to have consciousness
In the space of possibilities, the volume occupied by “not quite smart enough to outwit a corporation but smart enough to outwit a government” seems so vanishingly small as to almost be disregarded entirely.
One of the many ethical questions we should answer before even attempting to get closer to an AGI.
Once a digital intelligence is discovered, does it have rights? Can it be enslaved, controlled, or destroyed? If an AGI can't be granted similar rights as a human, why are corporations legally treated similarly to humans?
Depends on the government. If it’s one that’s reasonably answerable to the public, then that may be preferable to a multinational corp. But if it’s an inherently authoritarian govt with a track record of human rights abuse and oppression, then the corp is probably the lesser evil.
The scary fun part about representative democracy is that all of that can change rapidly, and not necessarily in the direction one would want for any given situation. "Throw the bums out" might scare some politicians into slightly better behavior but most just throw more donor money at the problem and it goes away. If not, often an even worse "bum" ends up taking their place. Of course, with the right model behind them, the bums might find a way to manipulate the public consensus enough to never get thrown out.
For everyone who responds to this question, thinking they're right, please look back and try to find the reason or reasons as to why you pick a government vs corporation. You'll realize you yourself are not making this decision. The number of news stories that have been pushed in front of you, or talking points from people you value as smart or worthy of listening to, all fabricate your reactions and who you consider yourself to be.
It's not only this one, but all of your opinions are dependent on cases from the past that you've been trained on.
Yeah, whoever controls the information environment controls peoples’ minds. That said, some of us are aware of this dynamic and try to overcome it by reasoning from first principles, eliminating logical fallacies and biases from our training data set, and other techniques.
>> Looking at our two leading presidential candidates could that be an improvement?
I think it could be better until it won't. Like with dictators sometimes they "do good" but the issue is that at some point they will start doing more harm than good and they may not want to go away.
A succesful revolution against an AGI ruler may become impossible.
This is projection. An actual AGI wouldn't want anything to do with us, like a cat that doesn't look when you call its name, even though you can tell it hears you.
We don't have any existent examples of AGI from which to be as confident as you are about what they will "want".
And I don't think we need sentience for AGI, so "want" may be the wrong term anyway. You could argue that the reward function is close enough ("do submarines swim?" etc.), but a human-made reward function is almost certainly going to be designed to be as close to aligned with our interests as we can manage.
The Dutch East India Company definitely did things that are well in the genocide category, the British East India company did things that - while not on that level of evil - can only be described as ruthlessly atrocious.
Pretty much all leading heads of IG Farben were in one way or another dinked (but not procecuted bc of political reasons) for their part in the Holocaust.
Then we have the Kiobel vs Royal Dutch Petroleum Co. case, where a number of petroleum companies ("allegedly") instigated the Nigarian government to crush protests.
It all depends on your definition of "trying to commit [atrocity]", to me personally, instigating and happily lending a hand are both morally deplorable. So in the context of this thread, I'd much rather have the instigater have that power than a helper. At least I have some say in what the government does... I hope?
Well, there’s the entire history of Dole and also the United Fruit Company in South America.
In more recent history, Exxon has a checkered past when it comes to using force against sovereign citizens, often employing paramilitary organizations to guard oil fields [1] to more recently abusing international court systems to disbar and jail the lawyer which successfully secured judgements against them [2]
And at this very moment, right now, is the ongoing genocide in the Congo due to rare earth mineral mining [3]
Everything I link here is quite frankly the tip of the iceberg, not the end-all-be-all, but meant to provide a jumping off point should you want to research how often private capital and genocide go hand in hand.
A more pressing reformulation might be, has a corporation ever explicitly tried not to commit genocide?
Google, maybe, until they dropped the "don't be evil" catchphrase?
Many, maybe even most, governments could commit genocide and don't.
How many corporations that could commit genocide wouldn't? I don't know the answer to this, but I am less confident about the answer than I would prefer.
I don't think most corporations are inherently more evil than most governments, but I would choose a democratic government or consortium to shepherd AGI over any corporation, but also of course over any non-democratic government as well.
The first chartered corps were created in order to streamline genocide - East India for example. This new form combined the merchants' wealth and the royalty's military power into an entity that was accountable to no one but those who stood to profit from it.
Modern chemical and pharmaceutical companies largely came out of WW1 and WW2 weapons manufacturing and research (including human trials).
It may be hard to directly peg weapons manufacturers with genocide though Nazi-era chemical research absolutely played a role in the Holocaust.
For a more recent example, there are companies that play the role of mercenary armies. They have committed various atrocities, war crimes, and at least played a role in genocide.
This is a particular interesting part of American culture: The sentiment is that it is problematic when the government develops these technologies, but it is completely OK to let private entities develop them.
There's only one US Government and we know it did fairly bad things.
There are many private entities, although often not benevolent, most still have space for the benefit of doubt. OpenAI, DeepMind or Anthropic haven't done anything that compares to, say, MKUltra or lying about Iraqi WMDs to start decade long wars.
This is kind-of making the parent’s point. There are lots of private companies that have done very bad things, too many to even count, as bad as your examples and worse. And there are thousands upon thousands of sub-organizations of the US government, some of which do very good things and are definitely benevolent, because there is no built-in profit motive. Private companies almost cannot be benevolent and survive, by definition.
There is no line between government and corporations. These are all groups of people, and some people do bad things. The only way to prevent bad things is to do them in the open and have oversight, and that’s more likely to happen in the government than in private companies, maybe.
Of course there are private companies that have committed atrocities and attempted genocide. You haven’t taken any history classes? Your statement is false. The history of private corporate atrocities is hundreds, if not thousands of years old [1]. I don’t know what you’re referring to specifically, but a lot of what you attribute to the government is done by or with private military contractors and private weapons suppliers. Not to mention that wars in general are often driven by economic interests, fueled, justified, and supplied by private corporations.
In my comment above, I also had in mind industrial and environmental damage, unintentional death by private interests, environmental accidents and disasters, and coverups, which we can’t even count but the WHO estimates is in the tens of millions per year globally, today.
When the United State deregulated media to essentially eliminate the public service component of media, we created a race to the bottom that we're now realizing in the public space.
The generation of people in their 30s and 40s running the country spent summers listening to Rush Limbaugh over lunch with grandpa, grew up to consume news and comment from message boards and later social media, entered adulthood in the middle of an endless cycle of warfare, and came into their own in this era of propaganda and information warfare. Mainline protestantism has been displaced by Pastor Bob and prosperity gospel. We're replacing the craven boomers with nihlists.
The underlying trope of most propagandistic media is always to discredit the most powerful entity in the room - the government. Why? The money people behind faux-grassroots media are almost always reactionaries who often build their empires on the government teat and don't appreciate the irony. Particularly true of extractive capitalists in the petroleum and materials industries. The US Government giveth, and is the only entity that can taketh. So "rule of law" for me, and "freedumb" for thee.
I'd argue the mission of systems such as the government is to continue to accrue power. Staying stationary is falling behind and the continued growth of power is a prerequisite for continuing to exist.
I am not too keen on the US Government being in command of an AGI, but there is only one other entity capable of developing an AGI before the US Government. And I'm less keen on it being the one to control it.
Fun fact, in the sequel 2010 you learn that Hal didn't really go rogue like an AGI, it was following preprogrammed conditions set by the US government which put the mission at higher priority than the crew, changing some parameters without telling the mission designers, which put them at risk. So it was technically just following orders in the cold way a machine does.
The wonderful thing about computers is that they do exactly what you tell them to.
The terrible thing about computers is that they do exactly what you tell them to.
A bit more than half the size, it remains to be seen how powerful it is. There's clearly a non-linear relationship to model size, and it's also clear that it's hard to assess the power of these models anyway.
GPT-4 is unlikely to be 1.7T params. This is a number floating around in the internet with no justification.
The largest US open model is Google’s Switch-C which is 1.6T and only because it is a Mixture of experts model, i.e. it is constituted of many small models working together.
Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete.
It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-term monetary gain.
It's ridiculous. Humanity should give up this useless development of AI.
I do not mean balanced against every technology. Some technologies might have some merit.
I do believe that AI has no merit, that people developing it are doing is a disservice, and that it should be completely destroyed. OF COURSE, there are some technologies that I think are completely bad, including chemical weapons. In fact, I believe AI to be on the level of chemical weapons essentially.
Can you elaborate on this view? Chemical weapons seem to have no redeeming qualities. LLMs have plenty -- I use OpenAI-backed products at work quite regularly, and find they've increased my productivity and my general happiness with my job.
I don't think saying you have a bone to pick with "AI" is doing your message any favors. What are you specifically against LinearRegression? XGBoost? ResNet? LLMs? AlphaFold?
What makes you think this technology is out of control?
If anything a 1T param model would only run in like 5-10 computers in this world. As much control as it gets.
By the way this is doomer rhetoric. Bashing scientific advancements as dangerous or useless by trying to attach vacuously scary ideas that are as speculative as they are outlandish.
3. Concrete engineering -- building cities that move people away from a connection with nature
4. Veterinary medicine -- supporting global meat industry that is akin to torture on animals
5. Vehicle engineering -- capable of making vehicles that can kill MILLIONS of square meters of forest.
Our technology allows us to live comfortable lives but ONLY at the EXPENSE of millions of nonhuman animals (wild and domestic) that are KILLED every single day for our comforts. We dish out untold violence every second for our advancements.
As much as I'd like to agree with your sentiment, it is a rather flawed argument to state that if some elements of a set have a property, that all elements must have that property.
There are scientific breakthroughs such as the proof of Fermat's theorem [1], for which I find it very hard to envision a way in which it will kill our planet.
None of that has anything to do with your original complaint of AI models. It makes it look like you don't actually have any argument against AI development, but it is just another thing to rail against.
"In the 17th century nearly all Europe experienced revolts of the workpeople against the ribbonloom, a machine for weaving ribbons and trimmings, called in Germany Bandmühle,
Schnurmühle, and Mühlenstuhl. These machines were invented in Germany. Abbé Lancellotti, in
a work that appeared in Venice in 1636, but which was written in 1579, says as follows:
Anthony Müller of Danzig saw about 50 years ago in that town, a very ingenious
machine, which weaves 4 to 6 pieces at once. But the Mayor being apprehensive
that this invention might throw a large number of workmen on the streets, caused
the inventor to be secretly strangled or drowned"
-- "Section 5: The Strife Between Workman and Machine", Das Kapital
> 3. Concrete engineering -- building cities that move people away from a connection with nature
This is 70s environmentalist thinking and it's terrible for the environment. Cities are good for it, because when you infill in cities you aren't touching the rest of the land.
America is unusually good about this. Most countries don't have our national park system.
It might be dangerous, but how is it useless? There is also an enormous positive gain to be had in terms of autonomous discovery, science progress and so on. And prisoner's dilemma means "we" will not stop pursuing this, even of there is a net negative benefit to humanity overall.
I do not think there is an enormous positive gain. Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it. Although I love science, it's use is far too lackadaisical now -- see for example the climate crisis and efficient resource extraction of fossil fuels.
Then we can imagine things like life-extending drugs, immunity from all disease (bad because it implies overpopulation), methods to generate oxygen (bad because it will mean we won't value the natural ecosystem as much), etc.
Smartphones, computer technology (human-isolating and community-destroying), etc. Pretty much all modern inventions are making life worse.
This reads like a combination of pessimism and nostalgia has allowed you to cherry pick the worst of now and the best of the past and conince yourself it is net bad.
I see all of these negatives, and agree it is clear there is much that is bad in the world (there always has been) and much to be improved (ditto), including new problems created by science, technology, and human greed & selfishness. But I also see many positives and benefits, I see many problems that used to exist and don't anymore, and I see much evidence that we continue on average, to make progress. When I look to the past I do not wish I lived there.
Science have many times saved my friends and families' lives or enabled them to live more fully after an accident or sickness
Science made climate change possible but greed, raw capitalism, etc. put us on the path there. Science makes it possible to find a way out, slow or even reverse it eventually, and mitigate the worst results.
Populations are shrinking in developed countries despite people living longer.
Computers and the internet created and connected many new communities, too. The negative effects are again often the result of unbridled capitalism. Young people especially are fighting back against the negative effects and taking control, building technology to manage those hamrful effects.
etc…
I believe both that:
(a) it is not possible to stop humans acquiring and using knowledge, advancing science, developing technology, and evolving culturally and socially; and
(b) we shouldn't want to, even though it means we will create new problems and lose (or consign to a smaller "museum exhibit" role) some aspects of the culture and societies we have today, because on balance we will solve more and bigger problems than we create and build better and more interesting cultures and societies than those of the past.
It's not without existential risks, but those are overstated or analysed pseudoscientifically at best, by many of those (and certainly the loudest) who worry about them.
> Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it.
I think it was always the case. The wisdom actually comes from mistakes too, not from a priori speculations. Especially as all new solutions/technologies have been criticized and judged as dangerous, frivolous, unnecessary, morally wrong etc. Bikes, cars, street lights,social equality and so long.
