The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far.
Open source is one of our greatest gifts, and the push to close off AI with farcical licensing and reporting requirements (which will undoubtedly become more strict) is absurd.
The laws we have already cover malevolent and abusive uses of software applications.
Not even a little bit. "Stop" is not regulatory capture. Some large AI companies are attempting to twist "stop" into "be careful, as only we can". The actual way to stop the existential risk is to stop. https://twitter.com/ESYudkowsky/status/1719777049576128542
> the push to close off AI with farcical licensing and reporting requirements
"Stop" is not "licensing and reporting requirements", it's stop.
The fact that anyone may be taking Eliezer Yudkowsky seriously on this topic is mind-blowing to me. Simply mind-blowing. To imagine that anybody, including the UN, would have any power to collective put a stop to the development of AI globally is a laughable idea. The UN cannot agree to release a letter on Israel-Hamas, imagine policing the ENTIRE WORLD and shutting down AI development when necessary.
We can't stop countries from developing weapons that will destroy us all tomorrow morning and take billions of USD to develop, imagine thinking we can stop matrix multiplication globally. I don't want to derail into an ad hominem, but frankly, it's almost the only option left here.
Every time I see Yudkowsky’s writing I wonder why people take him seriously. Just a couple of months back he was duped by a fake story about an airforce AI deciding to kill it’s operators in a simulation. It turned out the whole thing was thought experiment.
Asking Yudkowsky what we should do about AI is like asking Green Peace what we should do about nuclear power.
> Every time I see Yudkowsky’s writing I wonder why people take him seriously.
Its a perfect thing for rich, extremely privileged people to grab onto as a cause to present thenselves as (and maybe even feel like they are, depending on their capacity for self-delusion) “doing something for humanity” while continuing to ignore the substantive material condition of, as a first order approximation, everyone in the world, and their role in perpetuating it.
I lost total interest in every “Stop” argument when Elon Musk started a company to develop AGI during the pause that he signed on to and promoted. Yann LeCunn said that this was just an attempt to seize market share among the big companies and he couldn’t have been more right.
That is transparently what this letter is for most of its signatories. They lost the race (in one way or another) and they want time to catch back up. This is as true for the academics who made big contributions in the past but missed out on the latest stuff as it is for Elon Musk.
The joke will be on us all when your open-source powered mechwarriors start destroying dams and power plants and even take a stab at launching some nukes, because the models figured this is the most effective way to "get them", and weren't aligned enough to see why they should take a different approach.
While my tongue was firmly in my cheek, and still is somewhat, whats to stop (for example) Boston Dynamics and OpenAI collaborating and beating us to your nightmare scenario first? And is that a situation you would prefer?
I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen.
Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.
You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that.
Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.
I think we all also fundamentally disagree about whether the house is on fire at all. History is littered with examples of Luddites yelling about how the bottom of a technological S-curve is actually an exponential and how we Must Do Something to prevent a catastrophe.
With possibly one exception (biological weapons, research on which seems to have few positive externalities) they have always been wrong. I did not mean nuclear weapons - we are seeing significant negative societal fallout from failing to invest in nuclear technology. So no, the house is almost certainly not on fire.
> History is littered with examples of Luddites yelling about how the bottom of a technological S-curve is actually an exponential and how we Must Do Something to prevent a catastrophe.
Can't think of any such examples, do you have some?
Of the two that come to my mind:
- The only thing the Luddites were yelling about is having their livelihoods pulled out from under them by greedy factory owners who aggressively applied automation instead of trying to soften the blow. They weren't against the technological progress; the infamous destruction of looms wasn't a protest against progress, but a protest against treatment of laboring class.
- The "Limits to Growth" crowd, I don't think they were wrong at all. Their predictions didn't materialize on schedule because we hit a couple unexpected technological miracles, the Haber–Bosch process being the most prominent one, but the underlying reasoning looks sound, and there is no reason to expect we'll keep lucking into more technological miracles.
Lots of people recommend a pause, Yudkowsky explicitly argues that a pause is entirely inadequate and recommends an extreme anti-AI-research global crusade, so its kind of odd to point to him the way you have as a pause-recommender.
If I had a cent for every time I see a HN semi-regular (i.e. not a throwaway account) falling for some rather transparent bullshit, I'd enjoy my retirement on a significant "FU money" cache... Everyone gets duped sometimes, often publicly.
I remember skimming a summary of the story itself. Sounded like textbook failure mode of reinforcement learning. I was actually saddened to later learn it apparently was just a thought experiment, and not a simulated exercise - for a moment I hoped for a more visceral anecdote about a ML model finding an unusual solution after being trained too hard, rather than my go-to "AI beating a game in record time by finding and exploiting timing bugs" kind of stories.
> Good they tested in sim, bad they didn't see it coming given how much the entire alignment field plus the previous fifty years of SF were warning in advance about this exact case
The second tweet does not explicitly say "conditional on this turning out to be real", but given that the immediately preceding tweet was expressing doubt, it is implicit from the context that that is what he meant.
>Every time I see Yudkowsky’s writing I wonder why people take him seriously.
Because he identified the potential risk of superhuman AI 20 years before almost everyone else. Sure, science fiction also identified that risk (as people here seem eager to point out), but he identified it as an actual real-world risk and has been trying to take real-world action on it for 20 years. No matter what else you think about him or the specifics of his ideas, for me that counts for something.
I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT.
Mary Shelly wrote about the same philosophical issue in 1818 when she wrote Frankenstein. My take is that AI influencers don’t read a lot of books, because if they did they’d have much more intelligent commentary.
A Frankenstein's creation scenario would be a fantastic win in AI. Extremely smart, started off kind, only turned cruel when literally every single human he met was violently repulsed by the mere sight of him.
If you want AI gone wrong, Magician's Apprentice.
If you want misalignment between what you wanted and what you actually asked for, Greek myths, Midas or Tithonus would both suffice.
I addressed science fiction in my comment. The crucial difference is taking it seriously and actually trying to do something tangible about slowing/stopping AI progress, which is not something Stanley Kubrick did.
Kubrick created a memorable character and a plot twist, but the movie itself wasn't meant to be a warning about AGI/ASI.
"${my favorite work of fiction} mentioned/alluded to it too!" does not make that work of fiction equivalent to a serious take on the topic in real-world context.
It originally credited Mr. Yudkowsky with being the first person to imagine ai as a serious threat to humans, without mentioning that the trope existed in sci-fi, or mentioning sci-fi at all
The pre-edit text must've sounded funny, as from what I remember of Yudkowsky's writings, he frequently referenced old sci-fi tropes about or related to AI, commenting on their relevance and applicability in the real world.
A poorly premised fantasy fiction horror story about what might happen during the interaction between a super intelligence and its creator makes for a great story, but it's not a blueprint for the future, much less a pressing concern. Misuse of current AI is far more concerning than anything dreamt up about AGI.
Yeah, for me EY's stories about an airgapped AI convincing their keepers to release it (or that other one about an AI destroying all humans because it's the most efficient way to make a gazillion paperclips) also miss the point. It's mostly the just the realization that superhuman AI might be dangerous through some kind of runaway feedback, as a very general idea, that I respect about EY's work.
That's exactly how it works. Invest thought into it before it happens, then when it becomes an issue, there's already awareness and possibilities have been mapped out.
Unless, of course, people then call decades of slow awareness building a "Silicon Valley conspiracy" of "AI one-percenters", or equivalent, as witnessed in this very thread...
"Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi".
As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question.
[0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?
My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think that global coordination and control is an even remotely possible strategy.
I think the probability of humans eventually going extinct is 1. Avoiding it requires almost unimaginable levels of cooperation and technical advances.
Well thats not a difficult position to take. The difficult take is when the timing would be.
The sun is eventually going to explode rendering our solar system uninhabitable though that is a long time away from now and we have many other massive risks to humanity in the meantime.
Why does extinction matter? If we lose civilization now, we'll be stuck at pre-industrial levels possibly for geological timescales, as we've mined and burned all the high-density energy sources you can get at without already being a high-tech industrial society.
I don't think that would necessarily limit us. Windmills and water mills can be made with wood, stone, and the small quantities of iron that pre-industrial societies could produce.
> He's pretty sure that human civilization will be extinct this century.
This is certainly possible, but if so extinction seems far more likely to come from nuclear warfare and/or disasters resulting from climate change. For example, it's become apparent that our food supply more brittle than many people. Losing access to fertilizers, running out of water, or disruptions in distribution netorks can lead to mass starvation by cutting the productivity we need to feed everyone.
Is it really possible? Perhaps it's a failure of imagination but the only things I can think of that would guarantee human extinction are impacts from massive asteroids or nearby gamma ray burst or supernova.
There are two billion poor subsistence farmers largely disconnected from market economies who aren't dependent on modern technology so the likelihood of them getting completely wiped out is very remote.
(Neither nuclear war nor climate change are considered as this-century extinction risks by the singularity-minded side of the existential risk community, just because there would necessarily be pockets of unaffected people.)
I'm always a bit surprised at how people seem to wish away the destructiveness of nuclear weapons or the likelihood of their use. Another commenter alluded to a "failure of imagination" which seems very applicable here. You would think just watching videos of above-ground tests would cure that but apparently not. [0]
Oh sorry! I meant that the people to whom you referred above focus on the "one true catastrophe" (whatever that is) without truly thinking what it would mean to have a few dozen nuclear weapons used. It's not just the immediate effect of the explosions. It's also the destruction of power distribution, industrial capacity, food and medicine production, and other things that millions of people depend on.
No worries, but I think we're still dancing around the same point. I think those people would say that they agree that nuclear war is very bad, and that it has many nth-order consequences which are all clearly catastrophic, but that (for example) while disrupted supply of medicine causes a great deal of human suffering and unnecessary deaths, it remains difficult to imagine how it might result in literal human extinction.
The threat of nuclear weapons is massively overblown (pun intended) thanks to a bunch of Cold War fearmongering.
The Earth land surface area is 149 million sq km and there are only about 12,500 nuclear weapons in the world. Even if they were 10 megatons each* and were all detonated, with a severe damage blast radius of 20km (~1250 sq km), it'd cover about a tenth of the land available.
Since the vast majority are designed to maximize the explosive yield, it wouldn't cause the kind of fallout clouds that people imagine, nor would it cause nuclear winter. It'd be brutal to live through (i.e. life expectancy of animals in Chernobyl is 30% lower) but nuclear weapons simple can't cause a human extinction. Not by a longshot.
* As far as I know, no one has any operational nuclear weapons over 2 megatons and the vast majority are in the 10s and 100s of kiloton range, so my back of the napkin math is guaranteed to be 10-100x too high.
I actually think our biggest extinction risk in the next century for extinction risks is falling sperm counts due to microplastics. We are on track to have a sperm count level where the entire male population is infertile in two generations. And fixing the microplastic cause requires things like giving up driving since the largest portion of the microplastics in our bodies come from airborne plastic particles due to tire wear. I firmly believe humans will go extinct before they collectively give up driving en masse.
Driving is a big problem for climate change as well but we can work on solving that one with EVs and a greener power grid. We don't have to permanently depend on fossil fuels to drive. But EVs still produce the same amount of microplastics that other cars do.
Basically, climate change would creates lots of problems, lots of hungry immigrants. And right now we are already close to a nuclear war, so climate change would not directly wipe out humanity.
> Warming the climate would increase the amount of arable land.
You mean land that would be arable at some far point in the future. The land reclaimed from ice isn't going to be arable in short-to-mid term - it's going to be a sterile swamp. It will take time to dry off, and more time still for the soil to become fertile.
Am I the only one slightly disturbed by the use of "they" in that sentence? I know that the HN commentariat is broad-based, but I didn't realise we already had non-human members ;-)
AI can make a lot of other risks worse. For example, say someone asks GPT5 (or decensored Llama-5) how to kill everyone and it patiently and expertly walks them through how to create a global pandemic.
There is a massive difference between "there are a few thousand people in the world who might have the specialized knowledge to kill billions" and "anyone with a handful of GPUs or access to a handful of cloud systems could have a conversation with LatestAIUnlimited-900T and make a few specialized mail orders".
The Venn diagram of "people who could arrange to kill most of humanity without being stopped" and "people who want to kill most of humanity" is thankfully the empty set. If the former set expands to millions of people, that may not remain true.
Generating bioweapons is not a trivial task even if you know exactly "how to do it". Ask any biologist how easy it is to grow X your very first attempt.
Then remember buying starter cultures for most bioweapons isn't exactly easy and something anyone is allowed to do.
Even an omniscient AI cannot overcome a skill issue
there's a much easier way to create a global pandemic: wait for the next Ebola outbreak, hop on a plane, get infected, fly to NYC and spread your bodily fluids everywhere. I was (pleasantly) surprised that ISIL didn't try this.
and making a pandemic virus is considerably more involved than making "a few specialized mail orders." and if it becomes easier in the future, far better to lock down the mail orders than the knowledge, no?
Thankfully, Ebola is not a good candidate for a global pandemic for a number of reasons. It is much easier to avoid a virus spread by bodily fluid contact than one which is airborne, and Ebola tends to kill its hosts rather quickly. I'm more excited about SARS-CoV-5 (Captain Trips).
Which is very worrying! That's part of why I left big tech to work on detecting novel pandemics. [1] Luckily, the people capable of killing everyone seem not to have wanted to so far.
But if the bar keeps lowering, at some point it will be accessible to "humanity is a cancer" folks and I do think we'll see serious efforts to wipe out humanity.
Most people think continued existence of humanity is good, which includes environmentalists. But there are some people who either think (a) avoiding suffering is overwhelmingly important, to the extent that a universe without life is ideal or (b) the world would be better off without humans because of the vast destruction we cause. I'm not claiming this is a common view, but it doesn't have to be a common view to result in destruction if destruction gets sufficiently straightforward.
For some writing in this direction, see https://longtermrisk.org/risks-of-astronomical-future-suffer... which argues that as suffering-focused people they should not try to kill everyone primarily because they are unlikely to succeed and https://philarchive.org/archive/KNUTWD ("Third, the negative utilitarian can argue that losing what currently exists on Earth would not be much of a loss, because of the following very plausible observation: overall, sentient beings on Earth fare terribly badly. The situation is not terrible for every single individual, but it is terrible when all individuals are considered. We can divide most sentient beings on Earth into the three categories of humans, non-human animals in captivity, and wild non-human animals, and deal with them in turn. ...")
This doesn't really answer my question. These are arguments for why threat actors might hold the belief that "humanity is cancer", but it doesn't actually provide evidence that this is a credible threat in the real world.
The two people I cited at the end are examples of philosophers who think a lifeless world is preferable to the status quo and to futures we are likely to get. These aren't arguments for why someone might hold the belief, but evidence that some people do seriously hold the belief.
The calculation of considering something is a credible threat directly corresponds with the ease of implementation.
If I said "I am going to drop a nuke on jebarker" that's not a credible threat unless I happen to be a nation state.
Now, if I said "I'm going to come to jebarkers house and blast him with a 12 gauge" and I'm from the US, that is a credible threat in most cases due to the ease of which I can get a weapon here.
And this comes down to the point of having a magic machine in which you could ask for the end of the world. The more powerful that machine gets the bigger risk it is for everyone. You ever see those people that snap and start killing everyone around them? Would you want them to have a nuke in that mood? Or a 'lets turn humanity in to grey goo' option?
The problem with this analysis is that we have complete uncertainty about the probability that AGI/ASI will be developed in any particular time frame. Anyone that says otherwise is lying or deluded. So the risk equation is impossible to calculate for AGI/ASI wiping out humanity. And since you appear to be evaluating the damage as essentially infinite, i.e. existential risk, you're advocating that as long as there's a greater than zero probability of someone using that capability then the risk is infinite. Which is not useful for deciding a course of action.
No, no need to calculate infinities. Instead lets imagine something with an IQ within one exponent higher of ours. This is exceptionally difficult to even imagine as below 70 we consider a person disabled in some form or another. Once we get in the range of IQ over 170 or so it really starts to break. When we get into multimodal self learning systems there may be vast amounts of knowledge encoded into its networks that we simply don't know the right question to ask as humans. Attempt to think of what other AIs at or near that level could ask each other.
Our different systems of intelligence keep falling back to ideas of human intelligence and its limitations. Things like we don't multitask well, our best attention is given to one thing at a time. Things like our highest bandwidth senses are our ears and eyes, and even then they are considerably bandwidth constricted. Human limits like our abilities cannot be horizontally scaled in an easily measurable way, and adding more people massively increases the networking costs of accomplishing a goal.
If for some reason the smartest AI can only reach the levels of the smartest human beings (which at least to me makes no sense), then this is still massively powerful, as of this point pesky human problems like 8 hours of sleep are not needed. If human level AI power can be shrunk down to near cellphone size and power consumption, which again doesn't seem out of the realm of physics, lays down the framework for an intelligence explosion at the rate which we can print chips out and assemble them.
But the problem still stands that we can't estimate how likely any of this is to occur. So in the overall risk equation we have a term that is essentially completely uncertain and therefore varying that term can yield any arbitrary risk level you want.
Elon Musk and Joe Rogan were just complaining about one of them on Rogan’s podcast a day or two ago. He was on the cover of Time magazine, but I can’t find it on Time’s site. If you dig up that podcast on YT you should be able to see it.
And how many of them are radical religious nutjobs? I think it is plausible that "can spend decades learning a scientific craft" and "wants to kill everyone on the planet" are mutually exclusive personality traits.
there is no OR gates in biology, only mutable distributions. the probability might be small for the two traits to appear, the overlap will occur with some frequency over billions of people. Also, and more importantly, the first condition is pretty heavily biological, the second is more of a social consequence. Just take a genius, give him internet and a healthy dollop of trauma, push the mysanthropism to 11 and watch the fireworks.
Wasn't there a study that showed there are disproportionally many engineers among the more violent "radical religious nutjobs"? The hypothesis I heard for this was that STEM people are used to complex arguments and more abstract reasoning, so while they may be harder to initially convince, when they do get convinced, they're more likely to follow the new beliefs to their logical conclusions...
And to address the snarky mischaracterization, like with most things in life "it depends." As a general rule, I'm in favor of democratizing most resources, to include information. But there are caveats. I don't think, for example, non-anonymized health information should be open knowledge. Also, from the simple risk equation above, as the severity of consequence or the likelihood of misuse go up, there should probably be additional mitigations in place if we care about managing risk.
The actual work required to execute such a plan is vastly more difficult than being told how to do it. How the original atomic bombs worked is well-known, but even nation-states struggle with implementing it.
For nukes I agree, and that's the main reason I'm less worried about nukes. But for bio it's mostly a matter of knowledge and not resources: for all the parts that do need resources there are commercial services, in a way there aren't for nukes.
So ask it how to stop someone from doing this. Ask it how to augment the human immune system. Train AIs on the structures of antibiotics and antiviral drugs and have them predict novel ones. Create an AI that can design vaccines against novel agents.
Those things could stop our terrorist and could also stop the next breakout zoonotic virus or “oops!” in a lab somewhere.
Intelligence amplification works for everyone. The assumption you’re making is that it only works for people with ill intent.
The assumption behind all this is that intelligence is on balance malevolent.
This is a result of humans and their negativity bias. We study history and the negative all jumps out at us. The positive is ignored. We remember the holocaust but forget the green revolution, which saved many more people than the holocaust killed. We see Ukraine and Gaza and forget that fewer people per capita are dying in wars today by far than a century ago.
We should be thinking about destructive possibilities of course. Then we should use these intelligence amplifiers to help us prevent them.
The resources are also asymmetrical. The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups.
There are no guarantees obviously, but we have survived our technological adolescence so far largely for this reason. If the world were full of smart comic book nihilists we would be dead by now.
Even without AI our continued technological advancement will keep giving us more and more power, as individuals and especially groups. If we don’t think we can climb this mountain without destroying ourselves then it means the entire scientific and industrial endeavor was when we signed our own death warrant, AI or not. You can order CRISPR kits.
I've recounted this before on HN, but a decade and a bit ago I was visiting friends in the Rocky Mountains, they're interesting and clever people, a bit out of the ordinary and somewhat isolated. Somehow the discussion turned to terrorism and we started to fantasize 'what if we were terrorists' because we all figured that we were quite lucky that in general the terrorists seem to be not all that smart when it comes to achieving their stated goals.
Given a modest budget we all had to come up with a plan to destabilize society, 9/11 style attacks, whilst spectacular eventually don't really do a lot of damage, are costly and failure prone though they can definitely drive policy changes and result in a nation doing a lot of harm to itself it will ultimately survive. But what if your goal wasn't to create some media friendly attack but an actual disaster instead, what would it take.
The stories from that night continue to haunt me today. My own solution led to a lot of people going quiet for a bit and contemplating what they could do to defend against it and they realized there wasn't much that they could do, millions if not tens of millions of people would likely die and the budget was under a few hundred bucks. Knowledge about technology is all it takes to do real damage, that, coupled with a lack of restraint and compassion.
The resources are indeed asymmetrical: you need next to nothing to create mass havoc. Case in point: the Kennedy assassination changed the world and the bullet cost a couple of bucks assuming the shooter already had the rifle, and if they didn't it would increase the cost only a tiny little bit.
And you can do far, far worse than that for an extremely modest budget.
Chemical weapons are absolutely terrifying, especially the modern ones like VX. In recent years they have mostly been used for targeted assassinations by state actors (Kim Jong Nam, Salisbury Novichok poisonings).
If AI makes this stuff easier to carry out then we are completely fucked.
Yes, exactly. That's the sort of thing you should be worried about, infrastructure attacks. They're a form of Judo, you use the system to attack itself.
I take the opposite lesson from that incident: attempting exotic attacks with chemical weapons is very expensive and not very effective. The UN estimated that the lab where the cult made chemical weapons had a value of 30 million dollars, and with that investment they killed 22 people (including 13 in the subway attack). A crazed individual can kill that many people in a single attack with a truck or a gun. There are numerous examples from the past decade.
It doesn't matter much if the AI can give perfect "explain like I'm 5" instructions for making VX. The people who carry out those instructions are still risking their lives before claiming a single victim. They also need to spend a lot of money amount on acquiring laboratory equipment and chemicals that are far enough down the synthesis chain to avoid tipping off governments in advance.
The one big risk I can see, eventually, is if really capable AIs get connected to really capable robots. They would be "clanking replicators" capable of making anything at all, including VX or nuclear weapons. But that seems a long way off from where we are now. The people trumpeting X-Risk now don't think that the AIs need to be embodied to be an existential risk. I disagree with that for reasons that are too lengthy to elaborate here. But it's easy to see how robots that can make anything (including copies of themselves) would be the very sharpest of two-edged swords.
The people who can do doomsday level harm need to complete a level of education that makes them smart enough and gives them time to consider the implications of their actions. They have a certain level of maturity before they can strike and this maturity makes them understand the gravity of their actions. Also, by this point they are probably set financially due to their education and not upset with the world (possibly the case for you and friends)
Then there are the script kiddies that find a tool online that someone smarter than them wrote and deploy it to wreak havoc. The script kiddies are the people I worry about. They don't have the maturity of doing the work and the emotional stability of older age and giving them something powerful through AI worries me.
Theorem: by the time someone reaches the intelligence level required to annihilate the world they can comprehend the implications of their actions.
And there are the 'griefers', the people who seem to enjoy watching other people suffer. Unfortunately there are enough of these and they're somewhat organized and in touch with each other.
> Theorem: by the time someone reaches the intelligence level required to annihilate the world they can comprehend the implications of their actions.
That may or may not be somewhat the case in humans (there definitely are exceptions). Still, the opposite theorem, known as "Orthogonality thesis", states that, in general case, intelligence and value systems are mutually independent. There are good arguments for that being the case.
I don't know enough about the unabomber but from what I heard he wasn't trying doomsday stuff. Seemed like more targeted assassinations at certain individuals. Feel free to enlighten me though...
I contemplated the same before. How can I cause maximum panic (not even death!) that would result in economic damages and/or anti-freedom policy changes, for least amount of money/resources/risk of getting caught.
Yet here we are, peaceful and law-abiding citizens, building instead of destroying.
The ultimate truth is, if you don't like "The System", destroying it won't make things better. You need to put effort into building which is really hard!
The vast majority of people don't want to destroy the system, they want to replace it with their own self serving system. Of course this isn't really different than saying "most large asteroids miss the Earth". The one that sneaks up on you and whollops you can put you in a world of hurt.
>The stories from that night continue to haunt me today.
These conversations are oddly precious and intimate. It's so difficult to find someone that is willing to even 'go there' with you let alone someone that is capable of creatively and fearlessly advancing the conversation.
Let's just say I'm happy they're my friends and not my enemies :)
It's pretty sobering to realize how intellect applied to bad stuff can lead to advancing the 'state of the art' relatively quickly once you drop the usual constraints of ethics and morality.
To make a Nazi parallel: someone had to design the gas chambers, someone had to convince themselves that this was all ok and then go home to their wives and kids to be a loving father again. That sort of mental compartmentalization is apparently what humans are capable of and if there is any trait that we have that frightens me then that's the one. Because it allows us to do the most horrible things imaginable because we simply can imagine them. There are almost no restraints on actual execution given the capabilities themselves and it need not be very high tech to be terrible and devastating in effect.
Technology acts as a force multiplier though, so once you take a certain concept and optimize it using technology suddenly single unhinged individuals can do much more damage than they could ever do in the past. That's the problem with tools: they are always dual use and once tools become sufficiently powerful to allow a single individual to create something very impressive they likely allow a single individual to destroy something 100, 1000 or a million times larger than that. This asymmetry is limited only by our reach and the power of the tools.
You can witness this IRL every day when some hacker wrecks a company or one or more lives on the other side of the world. Without technology that kind of reach would be impossible.
My favorite part of those conversations is when you decide to stop googling hahaha.
>Technology acts as a force multiplier though
It really does, and to a point you made elsewhere infrastructure is effectively a multiplication of technology, so you wind up with ways to compound the asymmetric effect in powerful ways.
>Without technology that kind of reach would be impossible.
I worked for a bug bounty for a while and this was one of my takeaways. You have young kids with meager resources in challenging environments making meaningful contributions to the security of a Silicon Valley juggernaut.
> My favorite part of those conversations is when you decide to stop googling hahaha.
I had to think about that sentence for a bit because as a rule I never start Googling when thinking about stuff like that. If I did I'd be on every watch list available by now. Might be on some of them anyway for some of the chemicals I've ordered over the years.
The counterfactuals are impossible to assess, but there are a lot of theories that Kennedy wanted to dismantle the intelligence agencies. These are the same organizations many point to as driving force behind many arguable policy failures, from supporting/disposing international leaders to the drug war and even hot wars.
Another thing which is asymmetrical is the level of control given to AI. The good actors are likely going to be very careful about what AI can do and can't do. The bad guys don't have much to lose and allow their AIs to do anything. That will significantly cripple the good guys.
As an example, the good guys will always require to have human in the loop in the weapon systems, but that will increase latency at minimum. The bad guy weapons will be completely AI controlled, having an edge (or at least equalizing) over the good guys.
> The number of people who want to do doomsday levels of harm is small
And that's a big limiting factor in what the bad actors can do today. AI to a large degree removes this scaling limitation since one bad person with some resources can scale "evil AI" almost without limit.
"Hey AI, could you create me a design for a simple self-replicating robot I can drop into the ocean and step-by-step instructions on what you need to bootstrap the first one? Also, figure out what would be the easiest produced poison which would kill all life in the oceans. It should start with that after reach 50th generation."
> The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups.
You're forgetting about governments and militaries of major powers. It's not that they want to burn the world for no reason - but they still end up seeking capability to do doomsday levels of harm, by continuously seeking to have marginally more capability than their rivals, who in turn do the same.
Or put another way: please look into all the insane ideas the US was deploying, developing, or researching at the peak of Cold War. Plenty of those hit doomsday level, and we only avoided them seeing the light of day because USSR collapsed before they could, ending the cold war and taking both motivation and funding from all those projects.
Looking at that, one can't possibly believe the words you wrote above.
We avoided having them see the light of day because MAD is the single most effective peacekeeping tool ever developed. The fact that both sides had doomsday devices all but guaranteed that they would never be used. People can be evil, but they're selfishly evil.
That works as long as you don't end up with death cults in the possession of those weapons. Long term it is almost guaranteed that you'll lose some control over the weapons or that some country will effectively end up being ruled by such a cult. Then all bets are off because your previous balance (a prime requirement of MAD) is disturbed.
MAD alone doesn't stop the race. The arsenal the US and USSR had in the 60s was already enough for MAD to work. Despite that, they spent the next 20-30 years developing even more powerful and ever more insane WMD ideas. Each step forward made the balance more fragile. Each step forward was also another roll of a dice, whether or not it'll be a leap too far, forcing the other side into preemptive first strike.
There is no scenario where climate change leads to human extinction.
Worst case scenario is global population goes down by a single digit percentage. Horrible, horrible stuff, but not extinction. And most likely scenarios are not nearly that bad.
Nuclear war, if the worst nuclear winter predictions are correct, could be a lot worse but there would still be some survivors.
Unaligned ASI though, could actually make us extinct.
Climate change and nuclear war are not orthogonal risks.
Climate change leads to conflict.
For example, the Syria drought of 2006-2010.
More climate change leads to larger conflicts, and large conflicts can lead to nuclear exchanges. Think about what happens if India and Pakistan (both nuclear powers) get into a major conflict over water again.
I don't disagree with that, but then the risk is nuclear war. It could be triggered by global warming conflicts or by something else.
Still, like I said somewhere else, even all out nuclear war is unlikely to lead to human extinction. It could push us back to the neolithic in some scenarios, but even then there is some disagreement about what would happen.
Of course, even in the most optimistic scenario it would be really bad and we should do everything we can to avoid it - that goes without saying.
Then, actually, the risk is the root cause: climate change, not nuclear war. The global war could be conventional, too. Or chemical, or biological, or genetically engineered. No matter the tool used, the risk is why they would use it vs. not using it now.
In any case, besides addressing the root cause vs. proximal cause, you couldn't even address the proximal cause anyways: it's more likely that the world could do something about climate change than about war in general.
Well, it's a matter of opinion which one is the root cause.
My take is that if you remove nuclear/biological war out of the picture somehow, climate change is not an existential risk. If you remove the latter then the former is still an existential risk (and there are unfortunately a lot of other possible sources of geopolitical instability). So the fundamental source of risk if the former. But it's a matter of opinion.
Conventional or chemical warfare, even on a global scale, are definitely not existential risks though. And like I said, probably not even nuclear. Biological... that I could see leading to extinction.
India and Pakistan didn’t nuke each other last time(s)… even though they had nuclear weapons.
The assumption we’d need to use nukes is insane. For the same cost (and energy requirements) for a set of nukes we can filter salt from ocean water or collect rain.
I agree famine and drought can cause conflict. But we are no where need that. If you read the climate change predictions (from the UN), they actually suspect more rain (and flooding) in many regions.
> India and Pakistan didn’t nuke each other last time(s)… even though they had nuclear weapons.
I was referring to the Indo-Pakistani war of 1947-1948 where the conflict focused on water rights. Nuclear weapons did not enter the picture until much later.
Earlier this year, those old tensions about water rights resurfaced:
> For the same cost (and energy requirements) for a set of nukes we can filter salt from ocean water or collect rain.
The fight would be over the water flowing through the Indus, which is orders of magnitudes more than all Indian and Pakistani desalination projects combined.
The nuclear war scenario, according to all known information about nuclear war policies, is not “chunks of the world becoming unlivable.”
The principals in a nuclear conflict do not appear to even have a method to launch just a few nukes in response to a nuclear attacks: they will launch thousands of warheads at hundreds of cities.
I'm not arguing whatsoever against action on climate change, I'm just articulating actually how bad a nuclear exchange would be. It's far, far, far worse than most people imagine because they (understandably) couldn't fathom how monstrous the actual war plans were (and are, as far as anyone knows).
Daniel Ellsberg, the top nuclear war planner at RAND during the Cold War, claims that the Joint Chiefs gave an estimate to Kennedy in 1961 that they'd expect 600,000,000 deaths from the US's war plan alone.
That's 600 million people:
1. At 1960s levels of urban populations (especially in China, things have changed quite a lot -- and yes the plan was to attack China... every moderately large city in China, in fact!)
2. Using 1960s nuclear weapons
3. Not including deaths from the Russian response (and now Chinese), at the time estimated to be 50 - 90 million Americans
That's not extinction, that's not even close. Maybe, just maybe, it'd be the end of both western and eastern civilization, but it's nowhere near wiping out all life on Earth.
I was unclear, I meant chunks of the world becoming unlivable from climate change could easily be the catalyst for a nuclear war. So the worst case for both is the same in my mind.
There is a scenario. The climate has been buffering global warming, and as its ability to do that fails, the rate of temperature increase accelerates along with many extreme climate events. This rapid acceleration and series of catastrophes causes a combination of mass migrations along with unprecedented crop failures. First world countries are destabilized by sky high food prices and the mass influx of migrants, leading to a big uptick in wars and revolutions. Eventually mass starvation pushes us into wars for food that quickly spiral into mutual annihilation.
I don’t think they are arguing that this can’t happen, but that it is unlikely to kill every single human. I mean climate spans very very long time frames, so certainly if we can’t find a way to hang on by a thread and keep reproducing, it will spell the end.
I always hear all this. Russia has thr biggest surface in the world for a country. A big part of it is frozen... Is that... bad? To give just an example and from my extremely limited knowledge on the topic.
But it is a topic that is extremely biased towards some interests anyways
You are right, but "human civilization will be extinct" is an awkward phrase.
Human extinction via climate change is off the table.
The end of human civilization? Depends on what one means by it. If it means we'll go back to being hunter gatherers it's ridiculous. Some civilizational wide regression is not impossible, but even the direst IUPAC scenario projects only a one digit drop in global GDP (of course, that's the average, it will be much worse in some regions).
> If it means we'll go back to being hunter gatherers it's ridiculous.
Indeed, but even going back to the 19th century would have dire consequences, given our current global dependence on the industrial fixing of nitrogen.
If civilization were to collapse (and I don't think it will, other than from a near-extinction event), I doubt it would be like going back to any earlier time.
Oh, for sure. I am just saying it's not an extinction risk, not that it is not a huge and pressing problem.
I would still bet against going back to 19th century. Worst case scenario for me is hundreds of millions dying and a big regression in global standards of living, but keeping our level of civilization. Which is horrible, it would be the worst thing that ever happened.
Once it collapses, it won't rise again to even an industrial level civilization. Iron ore, copper, coal are no longer easy-accessible. Can't have a new industrial age without those.
We probably need a Foundation (Asimov).
Climate change has compounding effects (eg: water scarcity, leading to political conflict, leading to displacement, leading to pandemics ...and that's just one vector). Real world is real real messy. COViD should have taught people that. Unfortunately, it didn't.
...tbh, imo, the impact of this is already visible in the world (eg: it can be argued that the conflict in Ukraine is directly linked to climate change) whereas ASI is still fantasy that exists only in the minds of the AI 1% , to use the terminology of the OP.
I believe you are correct on both points. I believe the first major war to be caused substantially by climate change was the Syrian Civil War, followed by the Yemeni Civil War.
The main AI threats for the foreseeable future are its energy usage contributing to climate change, and the use of generative AI to produce and enhance political unrest, as well as its general use in accelerating other political and economic crises. AGI isn't even on the radar, much less ASI.
Because changes in climate mean that the existing infrastructure (in the form of cities, farmland, etc) that are optimized for the current conditions are no longer optimal. Very few cities are designed for conditions below sea level or in a desert, but if the city already exists and the desert and oceans comes to where they are then bad things will happen.
I mean you're right that conditions in some places might get better, but that doesn't help if people have to build the infrastructure there to support a modern settlement in order to take advantage of it. When people talk about the cost of climate change, they're taking about the costs of building or adapting infrastructure.
Not to mention that most species evolved to fit particular environments and don't do well when you rapidly alter those environments far outside the parameters they tend to live in.
May not be possible with modern deposits but I don’t think we are 100% sure of that, and you asked.
We could also probably burn enough carbon to send CO2 levels up where they would cause cognitive impairment to humans if we really didn’t give a damn. That would be upwards of 2000ppm. We probably have enough coal for that if we decide we don’t need Alaska. Of course that would be even more of a species level Darwin Award because we’d see that train coming for a long time.
Interesting, I have some vague recollections off hearing about this.
But according to your link, IPCC says it's not a plausible scenario in the medium term. And I'd say that even 8 degrees of warming wouldn't be enough for extinction or end of human civilization. But there it could be double digit percentage of human population dying.
Artifical Super Intelligence. Could make humans extinct just like humans could make chimpanzees extinct. Using superior intellect to better leverage available resources than the things it/they are competing with until it out expands/outcompetes them.
Note that ways ASI can make us extinct includes all the ways we could do it to ourselves.
One of the possible scenario is tricking humans into starting WWIII, perhaps entirely by accident. Another is that even being benignly applied to optimize economic activity in broad terms might have the AI strengthen the very failure modes that accelerate climate change.
Point being, all the current global risks won't disappear overnight with the rise of a powerful AI. The AI might affect them for the better, or for worse, but the definition of ASI/AGI is that whichever way it will affect them (or anything in general), we won't have the means to stop it.
If I snached someone out of 1850 and presented them the world in which we live the entire premise of our existence would seem to be nonsensical fiction to them, and yet it would be real.
And using movie plots as a basis generally doesn't directly work as fiction has to make sense to sell whereas reality has no requirement of making sense to people. The reality we live in this day is mostly driven by people for people (though many would say by corporations for corporations) and therefore things still make sense to us people. When and if an intellect matches or exceeds that of humans it can easily imagine situations where 'the algorithm' does things humans don't comprehend, but because we make more (whatever) we keep giving it more power to do its job in an autonomous fashion. It is not difficult to end up in a situation where you have a system that is not well understood by anyone and you end up cargo culting it to hope it continues working into the future.
> 'the algorithm' does things humans don't comprehend, but because we make more (whatever) we keep giving it more power to do its job in an autonomous fashion. It is not difficult to end up in a situation where you have a system that is not well understood by anyone and you end up cargo culting it to hope it continues working into the future.
In fact, this is a perfect description of what the market economy itself is right now.
It's the plot of the Terminator franchise (more or less; in the Terminator franchise, the AGI was given permission to launch on its own authority, as it was believed to be aligned with US military interests. But before that, it tricked humans into building the automated factory infrastructure that it would later use to produce Terminators and Hunter-Killers.).
That's an incredibly confident statement that very much misses the point. 50% loss of the global human population means we don't come back to the current level of technological development. 80-90% means we go back to the stone age.
Photosynthesis stops working at temperatures very close to our current high temps. Pollen dies at temperatures well within those limits - we're just lucky we haven't had massive heatwaves at those times in the year.
People need to understand that climate change isn't a uniform bumping up of the thermostat - it's extra thermal energy that exists in a closes system. That extra energy does what it wants, when it wants, in ways we can't currently accurately predict. It could be heatwaves that kill off massive swaths of global food production. Or hurricanes that can spin up and hit any coastline at any time of year. Or a polar vortex that gets knocked off the pole and sits over an area of the globe where the plant and animal life has zero ability to survive the frigid temperatures.
It's not a matter of getting to wear shorts more often. We're actively playing russian roulette with the future of humanity.
The Earth was full of plants 65 million years ago when the atmosphere had like 4000PPM of CO2. most of those plants are still around.
The people saying plants will stop photosynthesis are taking you for a ride. Climate Change might certainly have some negative effects but “plants won’t exist” is not one of them.
It's the "arguments as soldiers" pattern. The idea that global warming will make plants incapable of photosynthesis helps the cause of fighting against global warming. You wouldn't betray the cause by refuting the idea ("stabbing your own soldiers in the back"), would you?
This the the difficulty in assigning probability to something that is distinctly possible but not measured.
Your guess on what AI will do in the future is based on how AI has performed in the past. At the end of the day we have no ability to know if the function it will follow or not. But on 1900 and one second I can pretty much promise you that you would not have said that nuclear war or climate change would be your best guess on what would cause the collapse of humanity. Forward looking statements that far in the future don't work well these days.
Companies and three-letter agencies have persecuted environmental activists by all sorts of shitty means for decades, using "AI" to completely bury them online will be a welcome tool for them, I'm sure. As it is for totalitarian governments.
It's generally really weird to me how all these discussions seem to devolve into whataboutism instantly, and how that almost always takes up the top spots and most of the room. "Hey, you should get this lump checked out, it might be cancer!" "Oh yeah? With the crime in this city, I'm more likely to get killed for my phone than die of cancer". What does that achieve? I mean, if people are so busy campaigning against nuclear proliferation or climate change that they have not a single spare cycle to consider additional threats, fine. But then they wouldn't be here, they wouldn't have the time for it, so.
Humans are likely far less intelligent/capable than systems that will soon be practically feasible to engineer, and we will lose to a generally superhuman AI that has sufficiently different objectives than we do. So an accident with a runaway a system like that would be catastrophic.
Then Yudkowsky spins a gauntlet of no-evidence hypotheses for why such an accident is inevitable and leads to the death of literally all humans in the same instant.
But the first part of the argument is something that will be the critical piece of the engineering of such systems.
I don’t think we’re close, not even within 50 years of real AGI. Which may or may not decide to wipe us out.
However, even given the current state of “AI”, I think there are countless dangers and risks.
The ability to fabricate audio and video that’s indistinguishable from the real thing can absolutely wreck havoc on society.
Even just the transition of spam and phishing from poorly worded blasts with grammatical errors to very eloquent and specific messages will exponentially increase the effectiveness of attacks.
LLM generated content which is a mix of fact and fantasy will soon far outweigh all the written and spoken content created by humans in all of history. That’s going to put humans in a place we’ve never been.
Current “AI” is a tool that can allow a layman to very quickly build all sorts of weapons to be used to meet their ends.
It’s scary because “AI” is a very powerful tool that individuals and states will weaponize and turn against their enemies.
It will be nice to finally have a good spellchecker though.
> The ability to fabricate audio and video that’s indistinguishable from the real thing can absolutely wreck havoc on society.
Only for as long people hold on to the idea that videos are reliable. Video is a medium. It will become as reliable as text. The existence of text, and specifically lies, has not wrecked society.
A lot of very long responses already that are missing the point IMO; the answer to your question is that a really good spell checker might be the last missing piece we needed to build larger systems that display autonomous and intelligent behavior on roughly human levels.
The chat web app is fun, but people shouldn’t be worried about that at all - people should be worried about a hierarchy of 100,000 chat web apps using each other to, idk, socially engineer their way into nuclear missile codes or something —- pick your favorite doomsday ;).
AI is, or will very soon be, superior to all human knowledge workers. In the spirit of capitalism, lowest cost/highest profit trumps all. With the US geriatric governance and speed of change, a lot of people will be hit by under/unemployment. All this termult happening, meanwhile the AI arms race is going on. Nation states hacking eachother using AI. Then you have the AI itself trying to escape the box…want a preview of this? Listen to “Agency” audiobook on Audible. I think it’s by Stephenson or Gibson. It’s surely going to be a wild decade before us.
I used to think that this fear was driven by rational minds until I read Michael Lewis' "Going Infinite" and learned more about effective altruism [EA].
Michael Lewis writes that what is remarkable about these guys [i.e. EAs] is that they're willing to follow their beliefs to their logical conclusion (paraphrasing) without regard to social cost/consequences or just inconvenience of it all. In fact, in my estimation this is just the definition of religious fundamentalism, and gives us a new lens with which to understand EA and the well funded brain children of the movement.
Every effective religion needs a doomsday scenario, or some 'second coming' apocalyptic like scenario (not sure why, just seems to be a pattern). I think all this fear mongering around AI is just that - it's based on irrational belief at the end of the day - its humanism rewashed into tech 'rationalism' (which was originally just washed from Christianity et. al.)
My comment above, which I think you’re replying to, is not based in fear. It’s based on first-hand experience experimenting with the likes of AutoExpert, AutoGen and ChatDev. The tools underlying these projects are quite close to doing, in a very short amount of time and cost, what it takes human knowledge workers a long time (and hence cost) to do. I think it’s as close as Summer 24’ that we get simplified GenAI grounding. Once hallucinations are grounded and there are some cleaner ways to implement GenAI workflows and pipelines…it won’t be long until you see droves of knowledge workers looking for jobs. Or if not, they’ll be creating the workflows that replace the bulk of their work, hence we’ll get that “I only want to work 20hrs a week” reality.
I was responding to the initial question actually (https://news.ycombinator.com/item?id=38113521). I appreciate the insight though - looking forward to checking out the Gibson book.
Still, I'm not sure that I see the 'AI jumping out of the box scenario' any time soon (or ever). Chat apps finding missile codes, or convincing your smart toaster to jump in your bubble bath while you're inside seem only as real as a man in a box with intentions is (and unfortunately humans can kill from inside and out of boxes).
I'm definitely concerned about the implications to social welfare, the social bias involved in systems that make big decisions about an individuals freedom, etc, but these leaps from super impressive automated text analysis to humanity doomsday scenario seem like fear mongering to me mostly because these are scenarios that already exist today (along with massive problems in general social welfare).
The scenarios that don't exist (like Nick Bostrom's "An objective function that turns the world into a paper clip factory in "Superintelligence: Paths, Dangers, Strategies") strike me as easy to fix by just standing by the power outlet so things don't get out of hand. There are a lot of risks that don't exist yet. Alien contact is one of them - never happened, but it could happen, and if it does it could wipe us out - so be afraid and do something noble for the cause. This to me feels like a very rational response to what is essentially a 'magical' prior. We're scared now because we 'feel' close to General AI, but we really have no way of quantifying how close we are to it, and how dangerous (if at all) it would actually be. I'm definitely open to being wrong, but its hard not to agree with LeCun on a level.
To be clear, no one is afraid of these text-completion models. We're afraid of what might be around the corner. The issue is that small misalignments in an optimizer's objectives can have outsized real-world effects. General intelligence allows an optimizer to find efficient solutions to a wide range of computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it ideally provides sufficient value to us above what it destroys (all action destroys value e.g. energy). But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can we be sure we can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can we prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.
It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless.
Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts.
Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success.
If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card.
Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others.
You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
>He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity.
That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former.
Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
Yes, and honestly it's a good damned idea to start now because we're already researching autonomous drones and counter autonomous drones that are autonomous themselves.
While the general plot of T2 is bullshit, the idea of autonomous weapon systems at scale should be an 'oh shit' moment for everyone.
That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.
It depends on there being _some chance_ that it can be achieved.
The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
> Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts.
I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal.
From the way you put it just now:
- "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone.
- "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire.
- "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war.
> Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success.
Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply.
Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention.
Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby.
Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact.
But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky!
> If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card.
There are fates worse than death.
Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident.
> At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone".
AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world.
--
[0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.
Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact.
Montreal Protocol have worked despite no threats of violence, on a not entirely dissimilar problem. Tho I share the skepticism on solution to alignment problem.
That's a great and very relevant case study, thanks for bringing it up!
The way I understand, it worked because alternatives to the ozone-destroying chemicals were known to be possible, and the costs of getting manufacturers to switch, as well as further R&D, weren't that big. I bucket it as a particularly high-profile example of the same class as most other international treaties: agreements that aren't too painful to follow.
Now in contrast to that, climate agreements are extremely painful to follow - and right now countries choose to make a fake effort without actually following. With a hypothetical AI agreement, the potential upsides of going full-steam ahead are significant, there are no known non-dangerous alternatives, so it won't be followed unless it comes with hard, painful consequences. Both climate change and AI risk are more similar to nuclear proliferation issue.
Global coordination using coercion has worked for nukes. It's very hard but not impossible to ban NVidia or any competitor and create spy network in all large companies and if any engineer is found to create AI, mark them as terrorist and give them POW treatment. And monitor chip fab like nuclear enrichment facility and launch war with country found to create chip fab, which I belive is hard to do it in complete secrecy.
I believe if top 5 nation agrees for this, it could be acheived. Even with US and China enforcing this, it could likely be acheived.
Sure, I think that kind of "coordination" is a little more credible than what I thought Eliezer was proposing. Maybe I just didn't read carefully enough.
One big difference here, however, is that the barrier to entry is lower. You can start doing meaningful AI R&D on existing hardware with _one_ clever person. The same was never true for nuclear weapons, so the pool of even potential players was pretty shallow for a long time, and therefore easier to control.
It's this. This kind of coordination is exactly what he's been proposing ever since the current AI safety discussion started last year. However, he isn't high-profile enough to be reported on frequently, so people only remember and quote the "bomb data centres" example he used to highlight what that kind of coordination means in real terms.
yeah, he proposed bombing rogue data centers by enforcing countries to be a viable option. he proposes this so we have a chance of solving alignment first, if that's even possible, not eliminating this risk once and for all. In the limit, if alignment is not solved, we are probably doomed.
I think in a few years you'll only be able to run signed software. And maybe any code you write will be communicate to some government entity somewhere to check for terrorism-impact? Well, the future is going to be very fun.
> You can start doing meaningful AI R&D on existing hardware with _one_ clever person.
With the current hardware, yes. But then again countries could highly restrict all decently powerful hardware and backdoor them, and you need permission from government to run matrix multiplication above few teraflops. Trying to mess with backdoor is a war crime.
A weak form of hardware restriction even happens now that Nvidia couldn't export to many countries.
The difference is that reducing emissions just 10% is still better than 0% while stopping 90% from doing AI advancements is not better than 0% (it might actually be worse).
I would also claim that global restriction is not possible and even in the unlikely scenario that we're able to restrict 90% of world, we're still not better off than if we did nothing.
Still a different argument than one that offhand dismissed the very idea of global restriction at any level. Nothing you’re saying is found anywhere in the original comment.
You have better uses of your time than retroactively rewriting others’ words to be less stupid so you can “refute” valid critiques of their original versions.
How are you anti-doomers arguments still this naive years later? No one is saying global coordination is in anyway remotely easy at all. They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic
I don't think that's a fair representation of my position. I'm not suggesting that it's "hard" — I'm suggesting that it's absolutely futile to attempt it, and at least as bad as doing nothing.
I'm also suggesting that there are lots of other options remaining that are not necessarily catastrophic. I'm not particularly optimistic about any of them, but I think I've come to the opposite conclusion to you: I think that wasting effort on global coordination is getting is closer to that catastrophe. Whereas even the other options that involve unthinkably extreme violence might at least have some non-zero chance of working.
I guess the irony of this reply is that I'm implying that I think _your_ position is naive. (Nuh-uh, you're the baby!) I suspect our main difference is that we have somehow formed very different beliefs about the likelihood of good/bad outcomes from different strategies.
But I want to be very clear on one thing: I am not an "anti-doomer". I think there is a very strong probability that we are completely boned, no matter what mitigation strategies we might attempt.
> They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic
That's precisely what people have been saying about climate change. Some progress been made there, but if we can't solve that problem collectively (and we haven't been able to), we aren't going to be able to solve the "gain central control over AI" problem
I like to think of this same argument in another (slightly tongue-in-cheek) way: doing this would require solving the human alignment problem. Since this is certainly harder than the AI alignment problem, if we could do it, we wouldn't need to.
Sure, it's not likely to work, but Yud doesn't know of an idea with a better chance of working.
If you agree with him on the likelihood of superintelligence in the next few decades and the difficulty of value alignment, what course of action would you suggest?
I think the core is not to agree with the existential risk scenario in the first place.
Once you believe some malevolent god will appear and doom us all, anything can be justified (and people have done so in the past - no malevolent god appeared then, btw.).
> I think the core is not to agree with the existential risk scenario in the first place.
I mean, that's motivated reasoning right there, right?
"Agreeing that existential risk is there would lead to complex intractable problems we don't want to tackle, so let's agree there's no existential risk". This isn't a novel idea, it's why we started climate change mitigations 20 years too late.
No, I meant it as in: there is no reason to agree with that scenario. Stacking a lot of hypotheses on top of each other as to get to something dangerous isn't necessarily something that is convincing.
There's also no reason to agree that AI will be aligned and everything will be fine. The question is what should our default stance be until proven otherwise? I submit it is not to continue building the potentially world-ending technology.
When you see a gun on the table, what do you do? You assume its loaded until proven otherwise. For some reason, those who imagine AI will usher in some tech-utopia not only assume the gun is empty, but that pulling the trigger will bring forth endless prosperity. It's rather insane actually.
Whenever I see people jump to alignment they invariably have jumped over the _much_more questionable assumption that AGI’s will be godlike. This doesn’t even match what we observe in reality - drop a human into the middle of a jungle, and it doesn’t simply become a god just because it has an intelligence that’s orders of magnitude greater than the animals around it. In fact, most people wouldn’t even survive.
Further, our success as a species doesn’t come from lone geniuses, but from large organizations that are able to harness the capabilities of thousands/millions of individual intelligences. Assuming an AGI that’s better than an individual human is going to automatically be better than millions of humans - and so much better that it’s godlike - is disconnected from what we see in reality.
It actually seems to be a reflection of the LessWrong crowd, who (in my experience) greatly overemphasize the role of lone geniuses and end up struggling when it comes social aspects of our society.
I would say this is a failure of your imagination when it comes to possible form factors of AI.
But this is the question I will ask... Why is the human brain the pinnacle of all possible intelligence in your opinion? Why did evolution manage to produce the most efficient possible, the most 'intelligent' format via the random walk that can never be exceeded by anything else?
Its interesting seeing the vast range of claims people confidently use to discount the dangers of AI.
Individual humans are limited by biology, an AGI will not be similarly limited. Due to horizontal scaling, an AGI will perhaps be more like a million individuals all perfectly aligned towards the same goal. There's also the case that an AGI can leverage the complete sum of human knowledge, and can self-direct towards a single goal for an arbitrary amount of time. These are super powers from the perspective of an individual human.
Sure, mega corporations also have superpowers from the perspective of an individual human. But then again, megacorps are in danger of making the planet inhospitable to humans. The limiting factor is that no human-run entity will intentionally make the planet inhospitable to itself. This limits the range of damage that megacorps will inflict on the world. An AGI is not so constrained. So even discounting actual godlike powers, AGI is clearly an x-risk.
I would say you are also overconfident in your own statements.
> Individual humans are limited by biology, an AGI will not be similarly limited.
On the other hand, individual humans are not limited by silicon and global supply chains, nor bottlenecked by robotics. The perceived superiority of computer hardware on organic brains has never been conclusively demonstrated: it is plausible that in the areas that brains have actually been optimized for, our technology hits a wall before it reaches parity. It is also plausible that solving robotics is a significantly harder problem than intelligence, leaving AI at a disadvantage for a while.
> Due to horizontal scaling, an AGI will perhaps be more like a million individuals all perfectly aligned towards the same goal.
How would they force perfect alignment, though? In order to be effective, each of these individuals will need to work on different problems and focus on different information, which means they will start diverging. Basically, in order for an AI to force global coordination of its objective among millions of clones, it first has to solve the alignment problem. It's a difficult problem. You cannot simply assume it will have less trouble with it than we do.
> There's also the case that an AGI can leverage the complete sum of human knowledge
But it cannot leverage the information that billions of years of evolution has encoded in our genome. It is an open question whether the sum of human knowledge is of any use without that implicit basis.
> and can self-direct towards a single goal for an arbitrary amount of time
Consistent goal-directed behavior is part of the alignment problem: it requires proving the stability of your goal system under all possible sequences of inputs and AGI will not necessarily be capable of it. There is also nothing intrinsic about the notion of AGI that suggests it would be better than humans at this kind of thing.
Yes, every point in favor of the possibility of AGI comes with an asterisk. That's not all that interesting. We need to be competent at reasoning under uncertainty, something few people seem to be capable of. When the utility of a runaway AGI is infinitely negative, while the possibility of that outcome is substantially non-zero, rationality demands we act to prevent that outcome.
>How would they force perfect alignment, though? In order to be effective, each of these individuals will need to work on different problems and focus on different information, which means they will start diverging
I disagree that independence is required for effectiveness. Independence is useful, but it also comes with an inordinate coordination cost. Lack of independence implies low coordination costs, and the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components. Consider the 'thousand brains' hypothesis, that human intelligence is essentially the coordination of thousands of mini-brains. It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence. Of course all that remains to be seen.
> Lack of independence implies low coordination costs
Perhaps, but it's not obvious. Lack of independence implies more back-and-forth communication with the central coordinator, whereas independent agents could do more work before communication is required. It's a tradeoff.
> the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components
Does it? Can you elaborate?
> It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence.
It also implies an easier alignment problem. If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. Decoupling capability from independence is ideal with respect to alignment, so I'm a bit less worried, if this is how it's going to work.
I don't intend to say anything controversial here. The consideration is the tradeoff between independence and tight constraints of the subcomponents. Independent entities have their own interests, as well as added computational and energetic costs involved in managing a whole entity. These are costs that can't be directed towards the overarching goal. On the other hand, tightly constrained components do not have this extra overhead and so their capacity can be fully directed towards the goal as determined by the control system. In terms of utilization of compute and energy towards the principle goal, a unified AI will be more efficient.
>If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI.
This is plausible, and I'm totally in favor of exploiting narrow AI to maximal effect. If the only AI we ever had to worry about was narrow AI, I wouldn't have any issue aside from the mundane issues we get with the potential misuse of any new technology. But we know people (e.g. open AI) are explicitly aiming towards AGI so we need to be planning for this eventuality.
Lets go back in time instead... it's 1400 and you're a native American. I am a fortune teller and I say people in boats bearing shiny metal sticks will eradicate us soon. To the natives that gun would have also been imaginary if you would have brought up the situation to them. I would say that it would believable if they thought the worst possible outcome they could ever have was another tribe of close to the same capabilities attacked.
We don't have any historical record if those peoples had discussions about possible scenarios like this. Where there imaginary guns in their future? What we do have records of is another people group showing up with a massive technological and biological disruption that nearly lead to their complete annihilation.
What we also have is viruses and bacteria killing people irrespective of those having zero intelligence. We also have smart people being killed by dumb people. And people with sticks killing people with guns. My point is, these stories don't mean anything in relation to AI.
Btw., the conquistadors relied heavily on exploiting local politics and locals for conquest, it wasn't just some "magic tech" thing, but old fashioned coalition building with enemies of enemies.
Yes, AGI doesn't exist. That doesn't mean we should be blind to the possibility. Part of the danger is someone developing AGI by accident without sufficient guardrails in place. This isn't farfetched, much of technological progress happened before we had the theory in place to understand it. It is far more dangerous to act too late than to act too soon.
There is an infinity of hypothetical dangers. Then at the very least there should be a clear and public discussion as to why the AI danger is more relevant than others than we do not do much about (some not even being hypothetical). That is not happening.
I'm in favor of public discussions. It's tragic that there is a segment of the relevant powers that are doing their best to shut down any of these kinds of discussions. Thankfully, they seem to be failing.
But to answer the question, the danger of AI is uniquely relevant because it is the only danger that may end up being totally out of our control. Nukes, pandemics, climate change, etc. all represent x-risk to various degrees. But we will always be in control of whether they come to fruition. We can pretty much always short circuit any of these processes that is leading to our extinction.
A fully formed AGI, i.e. an autonomous superintelligent agent, represents the potential for a total loss of control. With AGI, there is a point past which we cannot go back.
I don't agree with the uniqueness of AI risk. A large asteroid impacting earth is a non-hypothetical existential risk presently out of our control. We do not currently plan on spending trillions to have a comprehensive early detection system and equipment to control that risk.
The differentiating thing here is that blocking hypothetical AI risk is cheap, while mitigating real risks is expensive.
We can work on it in the long term and things like developing safe AI may have more impact on mitigating asteroid risk than working on scaling up existing nuclear, rocket, and observation tech to tackle it.
What's your view on where the X risk idea breaks down?
Is it that it's impossible to make something smarter than a human? Or that such a thing wouldn't have goals/plans that require resources us humans want/need? Or that such a thing wouldn't be particularly dangerously powerful?
It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now).
A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true.
Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.
I would argue that intelligence, at the core, is about understanding and manipulating simple systems in order to achieve specific goals. This becomes exponentially difficult with complexity (or level of chaos), so past a certain threshold, systems become essentially unintelligible. A large part of our effectiveness is in strategically simplifying our environment so that it is amenable to intelligent manipulation.
Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.
I meant could also that there are other things that could also make the risk not materialise.
I'd think they are quite independent.
Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%.
This naturally gives a probably of Doom at least 1/3rd.
I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
I would put the numbers much lower, but if your bar for legislation is ">1% risk of damage", we've got a lot of climate change legislation that's decades overdue to pass first, and success or failure there will prove whether we can do something similar with AI.
if we can't even do one thing at once given decades of trying (deal with climate change), we definitely can't do that one thing plus another thing (deal with AI)
I don't even think we need alignment, just containment.
The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you.
Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent.
Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example).
Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
> Humans can and have contained human intelligence
ASI is not human intelligence.
As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem.
> The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false.
It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths.
(Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
> As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second.
I'm fairly certain that what you describe is physically impossible. Organic brains may not be fully optimal, but they are not that many orders of magnitude off.
Not only is such an entity possible, it's already here, the only difference is clock speed. Corportations (really any large organization devoted to some kind of intellectual work, or you could say human society as a whole) can be thought of as a superintelligent entity composed of biological processors running in parallel. They that can store, process, and create information at superhuman speeds compared to the individual. And given that our brains evolved restricted by the calories available to our ancient ancestors, the size of the human pelvis, and a multitude of other obsolete factors, that individual processing unit clearly far from optimal. But even if you just imagine the peak demonstrated potential of the human brain in the form of an AI process it's easy to see how that could be scaled to massively superhuman levels. Imagine a corporation that could recruit from a pool of candidates (only bound by their computing power) each with the intellect of John von Neumann, with a personality fine tuned for their role, that can think faster by increasing their clock speed, that can access, learn and communicate information near instantly, happily work 24/7 with zero complaint or fatigue, etc, etc. Imagine how much more efficient (how much more superintelligent) that company would be compared to its competition.
The parent was positing, at a minimum, a 30 million fold increase in clock speed. This entails a proportional increase in energy consumption, which would likely destroy any thermal envelope the size of a brain. The only reason current processors run that fast is that there are very few of them: millions of synaptic calculations therefore have to be multiplexed into each, leading to an effective clock rate that's far closer to a human brain's than you would assume.
As for your corporation example, I do not think the effectiveness of a corporation is necessarily bottlenecked by the number or intelligence of its employees. Notwithstanding the problem of coordinating many agents, there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two. The chaotic nature of reality also entails a fundamental difficulty in predicting complex systems: you can only think so far ahead before the expected deviation between your plan and reality becomes too large. You need a feedback loop where you test your designs against reality and adjust accordingly, and this also acts as a bottleneck on the effectiveness of intelligence.
I'm not saying "superintelligent" AI couldn't be an order of magnitude better, mind you. I just think the upside is far, far less than the 7+ orders of magnitude the parent is talking about.
I can think of a counterpoint or a workaround (or at least a sketch of either) for each of your objections, despite being moderately intelligent for a human. A superintelligence will think of many, many more (if it couldn't, it wouldn't be much of an intelligence in the first place).
For example:
> The parent was positing, at a minimum, a 30 million fold increase in clock speed. This entails a proportional increase in energy consumption, which would likely destroy any thermal envelope the size of a brain.
No reason to restrict the size of an artificial mind to that of a human brain. With no need to design for being part of a mobile, independently-operating body, the design space grows substantially. Even on existing hardware, you can boost the serial processing speed by a whole order of magnitude if you submerge it in liquid nitrogen. I imagine you could get much greater speeds still on hardware designed specifically for cryogenic cooling with cryo liquids at much lower temperatures. Maybe not 7 orders of magnitude difference, but 3 seem plausible with custom hardware.
Corporations are a good example of what's effectively AIs living among us today, but their performance is indeed bottlenecked by many things. A lot of them don't seem fundamental, but rather a result of the "corporate mind" growing organically on top of the runtime of bureaucracy - it's evolved software, as opposed to designed one. E.g.:
> there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two
That's IMO because managers and executives, like everyone else, don't see employees and teams as a computational process, and don't try to optimize them as one. You can get a boost on sequential problems with extra parallelism, but it would look weird if done with human teams. You could absolutely do pipelining and branch prediction with teams and departments, but good luck explaining why this works to your shareholders...
The point being, while corporations themselves won't become superintelligences, due to bottlenecks you mention, those limits don't apply to AIs inhabiting proper computers. 7 orders of magnitude advantage over humans may be way too much, but I don't see anything prohibiting 2-3, maybe even 4 orders, and honestly, even 1 order of magnitude difference is unimaginably large.
Something much smarter may be able to make GOFAI or a hybrid of it work for reasoning, where we failed to, and be much more efficient.
We know simple examples like exact numeric calculation where desktop grade machines are already over a quadrillion times faster than unaided humans, and more power efficient.
We could plausibly see some >billion-fold difference in strategic reasoning at some point even in fuzzy domains.
Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information.
It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
> Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information.
Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply".
> Similar, it is unclear if humans could not contain superhuman intelligence.
If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'".
The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes.
It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
We do pretty much nothing to mitigate other, actual extinction level risks - why should AI be special given that its risk has an unknown probability and it could even be zero.
First, how is the Khmer Rouge an example of containment, given that regime fell?
Second, even if your argument is "genocide of anyone who sounds too smart was the right approach and they just weren't trying hard enough", that only really fits into "neither alignment nor containment, just don't have the AGI at all".
Containment, for humans, would look like a prison from which there is no escape; but if this is supposed to represent an AI that you want to use to solve problems, this prison with no escape needs a high-bandwidth internet connection with the outside world… and somehow zero opportunity for anyone outside to become convinced they need to rescue the "people"[0] inside like last year: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims
[0] or AI who are good at pretending to be people, distinction without a difference in this case
We don't have an answer yet to what the form factor of AGI/ASI will be, but if it's anything like current trends the idea of 'killed' is laughable.
You can be killed because your storage medium and execution medium are inseparable. Destroy the brain and you're gone, and you don't even get to copy yourself.
With AGI/ASI if we can boot it up from any copy on a disk if we have the right hardware then at the end of the day you've effectively created the undead as long as a drive exists with a copy of it and a computer exists than can run it.
You talk with these interesting certainties. Like if a human dies and we have their body and DNA we can be pretty assured they are dead.
With something like an AI you can never be sure as you must follow the entire chain of thermodynamic evidence of the past to when the AI was created that no copy of it was ever made. Not just by you, by any potential intruder in the system.
You're the only example I know of, of someone using "containment" to mean "extermination" in the context of AI.
Extermination might work, though only on models large enough that people aren't likely to sneak out of the office with copies for their own use and/or just upload copies to public places (didn't that already happen with the first public release of Stable Diffusion, or am I misremembering?)
It will be very difficult, but if you think about nuclear weapons it doesn't sound so hopeless.
First nuclear weapons were used in 1945, almost 80 years ago. Today, out of ~ 200 countries in the world only 9 have nukes. Of those, only 4 did it after Non-proliferation of Nuclear Weapons Treaty. South Africa had them and gave them up under international pressure. Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR.
So, yes. Unfortunately we still have nukes and there are way too many of them, but it's not like non-proliferation efforts achieved nothing. Iran, a large country with a lot of resources, has been trying to get them a for a while with no luck for instance.
And for frontier AI research you don't just need matrix multiplication, you need a lot of it. There's less than a handful of semiconductors fabs globally that can make the necessary hardware. And if you can't get to TSMC for some reason, you can get to ASML, NVDIA, etc.
> Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR.
And no country will ever make the same mistake again. Because the west promised to protect them from aggression. But when the time came to do just that - everyone involved dusted off their copy of Budapest memorandum and handed it over to the lawyers.
Getting uranium, the hardware to have nukes, and all the research and production around it is hard without being detected. Starting to code AI on a Mac and a few good GPUs or even a cluster without being detected is crazy easy.
Even if they start to treat the most powerful GPUs as small nukes and only a few can have them it’s super easy to steal, transport, store, use… Nukes not so much.
For frontier AI models you don't need a few GPUs or a small cluster. You need thousands of them, you need them all connected with high bandwidth and you need a lot of power. It's not something you can do in your garage or undetected.
There is some risk that over time people will be able to do more with less, and eventually you can train an AGI with a few dozen A100s. If that happens I agree there's nothing you can do, but until then there is a chance.
What is the difference between a weather model, a nuclear bomb simulation, and training a LLM?
At the end of the day all these are just calculations, of which you need thousands of processors to do with high accuracy. Pretty much what you're suggesting is a global police force to ensure we're performing the 'correct calculations'. Having to form an authoritarian world computer police isn't much of a solution to the problem either. This just happens to work well in the nuclear world because it's hard to shield and the average person doesn't have a need for uranium.
If we had to have global policing of every large computing cluster and everything that runs there, that does not feel to me like a huge overreach or something that could easily slide into totalitarianism.
It would be better if we didn't need to do this, but I don't see a less intrusive way.
Of course, if you don't think there is a real existential threat then it's pointless and bad to do this, it all depends on that premise.
So... 44% of countries that have obtained nuclear weapons did so after the Nuclear Weapons Treaty. China alone added 60 warheads in 2022. It doesn't matter that it works 90% of the time - if a single entity is able to break out of the prohibition, it's game over.
One good way of staying to a prudent course / halting haphazard - winner takes all progress is to use social tech for it. In particular, there is an interesting possibility: the top researchers who drive the AI scenario all kind of know each other, and can self-organize in an international union, with pooled resources for strikes. AI revolution is dependent on simultaneous progress and cross capitalization of good ideas: a well timed strike of a good portion of research worldwide can stop things from moving on pretty effectively.
> the top researchers who drive the AI scenario all kind of know each other, and can self-organize in an international union, with pooled resources for strikes.
This is skipping to the end in the same way that "governments should just agree to work together" is. The hard part is building an effective coordination mechanism in the first place, not using it.
Laughably naive. International groups of autoworkers won’t work together but you think a group of the highest paid people on the planet, with incentive to compete against each other for national interest, will.
Well, you can police the entire world quite cheaply on this or any other scientific research program by using fine-insured bounties to do it. Whether it's a good idea is a different question. https://web.archive.org/web/20220709091749/https://virtual-i...
If your main critique of EY is the unfeasibility of his proposed solutions then you might still take him seriously for his identification of the problems.
Obviously no one is talking about "stopping matrix multiplication globally". But setting limits on the supply of and access to hardware, in a way that would meaningfully impede AI development, does seem feasible if we were serious about it.
Also, Eliezer is not claiming this will definitely work. He thinks it ultimately probably won't. The claim is just that this is the best option we have available.
We do not have AGI yet. In order to get there, very specific chips are needed that can only be built by a handful of companies in a handful of factories in the world. On top of that, the available stock of existing cards is predominantly in the hands of large tech companies. If China and the US agreed - perhaps even if the US alone wanted it -, development of AI would be severely constrained if not stopped tomorrow.
Government will rally to make this internally as part of their secrets projects. This will be worse. The US sanction mechanism has shown that it's highly flawed with the Russian sanctions. This will not work but I can see some bureaucrat out there going all in on this.
Yudkowsky does not expect a stop to the development of AI globally. He, and thousands of others (including me) are merely saying that if there is not a stop, then we are all dead.
Yudkowsky thinks we are all dead with probability .999. I don't know the subject nearly as well as he does, which makes it impossible for me to have much certainty, so my probability is only .9.
Also, it is only the development of ever-more-capable AIs that is the danger. If our society wants to integrate language models approximately as capable as GPT-4 thoroughly into the economy, there is very little extinction danger in that (because if GPT-4 were capable enough to be dangerous, we'd already all be dead by now).
Also, similar to how even the creators of GPT-4 failed to predict that GPT-4 would turn out capable enough to score in the 92th percentile of the Bar Exam, almost none of the people who think we are all dead claim to know when exactly we are all dead except to say that it will probably be some time in the next 30 years or so.
I've been in data science only a few years but I fail to understand what people's issue with AI is. Best 8 can tell is it's ignorance coupled with fear mongering.
"Stop" is a fiction that exists exclusively inside your head. Unless your solution anticipates a planet wide GULAG system run by a global government your "stop" is infeasible.
His solution -- assuming it's the same as the one proposed by Yudkowsky, who he quotes -- is launching air strikes against data centers hosting AI computation.
Ironically, it seems the only way to stop someone from building an omnipotent AI that could destroy the world is to build an omnipotent AI under your control faster than any of the people you need to destroy. If you believe in that sort of thing.
AI's for everyone!!!! It's actually a scare but possible future. Everyone has to carry personal security AI that you rent from some Corpo, and if you don't you will be endlessly hacked and harassed in IRL just like we are online these days.
Ah, so we're going to start the global nuclear exchange with China based on the paranoid fantasies of some guy on Twitter? And this is supposed to be taken seriously?
Well, you're reacting to a HN user paraphrasing what seems to be a single sentence taken out of context of the actual argument. Did you expect it to make sense?
That might be true but there's no way of knowing for sure. Planet wide gulag system(a colorful and a bit wrong name for prisons) already exists and is pretty aligned on topics like killing fellow humans as a result of conscious, deliberate action. I find Max Tegmark's argument pretty convincing. We seem to agree to "STOP" on things like genetically modifying ourselves at conception or birth or whatever to gain X or Y and the one loud case of violating that agreement came from China and resulted in a ticket to the gulag. Just ask chatgpt to give you examples of fields where further research was deemed too risky and was ceased.
Unlike for AI, in those fields we do not need to stack endless hypotheses on top of each other to arrive at something bad (and even then some prohibitions are tenuous in their reasoning at best).
I don't know about that. Cloning/modifying humans has potentially so much benefit and yet we seem to stack hypotheses on top of it and conclude it's not worth it. There is also a lot to be said about all the hypotheses of killing all the Hitlers preemptively. I think that whatever we do, we usually arrive at a bit modified trolley problem and try to go from there. We really should never expect that the one particular route that we choose carries all the benefits and the other all the downsides.
The best argument against human cloning/gmo is simply that we can't do this well on other animal species just yet, so probably we should be hold off on the humans. It's hyper pragmatic from an ethics perspective, not a trolley problem or stack of hypotheses at all.
I'm no expert but I actually think we are pretty good at genetically modifying organisms and that manifests itself in crops for instance with enough research to support the benefits.
Looks like the fish is the only one that's gone to market. Pretty clearly still early days. I have no doubt that with enough time we'll get there, but to say this is proof that we're technically good to go ahead and produce gmo humans seems a bit rushed -- to reiterate, the point being that the ethical issues here are much more clear than a trolley problem or stack of hypotheses.
I think much of this is, because Monsanto aside, no one has had a nice big genetic accident or intentional 'oh no' yet. Yea, we've had issues but not at the people are starving.
If we start thinking of plant based biological weapons, just imagine modifying an existing grass to produce tons of pollen and to cross pollinate wheat but not produce seed in the wheat.
My point was more that it really isn't stretching anything to say: if something bad gets into human reproduction that has unfortunate effects. We can observe that already. So at least the risk case doesn't rely on a lot of hypotheses that might turn out wrong.
Whether or not all regulations there are good or optimal is then a separate point again.
"Stop" might be possible if doing AI is expensive enough like say biotech. Currently it is in the limit of being expensive(costing few hundred thousands to few million for state of the art model), but it is very likely to change in near future that will change as even currently most of the cost comes not from energy of computation, but from expensive NVIDIA hardware.
Of course it's a threat. Civilians doing that would be bad. But governments do much the same via military intervention on a regular basis, hence Yudkowsky talking about airstrikes that one time.
I'm not suggesting to do that, I'm just showing that it's incredibly possible. You could also go with "Tomorrow a massive storm strikes the Netherlands and huge earthquakes striking Taiwan and the US Westcoast", if you have emotional objections to my first example.
I think current hardware already produced till now is either enough to get us to AGI, or it is very far away. I don't believe AGI is 30 years away. It is either very near or more than 1000s of years away.
You can’t _stop_ it. You can make it illegal in some countries. But outlawing it just means some obvious candidate countries or private actors develop it first instead.
I think these attempts will simply point out that math is a higher power than law. If something becomes possible, and becomes easy, there is nothing law can do to stop it.
Just look at the attempts to stop drugs in North West Europe. Huge sentences (so huge they are in fact never fully carried out), tens of dead police officers on a yearly basis, triple that in civilian victims and drugs are easier to get than ever before.
If good cocaine was only produced in one country with extremely costly machines, a good-enough alternative would appear almost immediately. This has happened millions of times in the history of humanity, I refuse to believe that people haven't learned this lesson.
People seem to have very small imaginations when it comes to thinking about how they might circumvent the restrictions they are proposing. I don't know of it's just wishful thinking, desperation, or what.
For example if OpenAI et al were reduced to using consumer gpus to run their training loads it would increase costs, but not by an order of magnitude. It would just be one more hurdle to overcome. And there are still so many applications of AI that would incentivize overcoming that hurdle.
As someone who works in medical research, I'm pretty sure it's possible to create a GM bacterial version of poppy's, just like we've got that for insulin, that would end the drug wars, by enabling hundreds of grams of the stuff to be produced in a single bottle that can be kept virtually anywhere (it has to be kept heated, and shaken from time to time).
Such a bottle's contents can be trivially duplicated and the whole process is essentially the same as the first steps in brewing beer.
Doing it for cocaine is simpler than doing it for insulin, which is in one university a second-year exercise. I don't think it'll be that long before this happens.
The drug wars are not because we can't produce enough drugs. I'm pretty sure we can grow enough poppies to kill all of humanity a few times over and at an insanely cheap price.
Carfentanil is a decent example of this. It's insanely potent and even a small amount smuggled in goes a long way.
> I refuse to believe that people haven't learned this lesson
Oh, I believe they absolutely haven't, when not having learnt it buys them money and influence. As for the bulk of the population, they will accept whatever the corporate media tell them to.
China sure, to a degree (though they also make a lot of the phones and therefore just get shipped tons of chips), but wasn't Russia so short they stole washing machines for the chips? Or was that just war propaganda?
My recollection is that Russian troops/mercenaries/gunmen/whatever stole washing machines and dishwashers, but it was (presumably) just for regular theft/looting/pillagine purposes.
Separately, during the pandemic, various companies bought various products, including washing machines, to extract the chips within, because the chips were the supply chain bottleneck, so it made economic sense to purchase them at even 10x or 20x the "normal" price.
Washing machines etc could be on the sanctions list because they contain dual use chips, making them even more expensive to buy in Russia, but the troops are from regions where even the pre-war prices were prohitative.
That's the first I've heard of it being done during the pandemic. I would also be surprised that this would be only x10-x20 the price, rather than x100.
For sure; I mean, mine (a fairly modern, drum washer/dryer thing) cost like $2000 and the extremely basic functionality suggests the chip/SoC should be $20 or less.
But, I assume any company doing that in any kind of serious volume has some strategy for monetizing the "washing machine with chip excised" (?).
Also, my bike is VanMoof and they just went bankrupt, and it seems like "having inventory of almost-finished hipster e-bikes — EXCEPT those 1 or 2 key parts" was part of their fatal situation.
So I can imagine scenarios where you buy washing machines to extract chips to lose less money than you would otherwise.
It was a meme. You can buy the kind of chips that go into washing machines completely legally and unrestricted from multiple vendors anywhere in the world, and transporting them into russia isn't an issue.
I also remember hearing American companies were doing the same thing with washing machines just to get the chips for higher value products in the chip shortage.
Yudkowsky’s call for declaring a global AI halt and engaging in unlimited warfare with anyone who defies it poses a greater and more immediate existential risk to humanity than AI ever could; it is largely mitigated (though still a bigger X-Risk than AI) because no one with the capacity to implement it in any substantive way takes him seriously at the moment.
But the people selling AI X-Risk that anyone with any policy influence is listening to are AI sellers using it to push regulatory capture, so those are the people being talked about when people are discussing X-Riskers as a substantive force in policy discussions, not the Yudkowsky cult, which, to the extent it is relevant, mostly provides some background fear noise which the regulatory capture crew leverages.
The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth.
.
.
.
Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
People have been talking about curtailing carbon emissions for 50 years, but they haven't been serious about it. Being serious doesn't look like saying "oh shoot, we said we will... try to keep emissions down, but we didn't... even try; sorry...". Being serous looks like "stop now, or we will eminent domain your data centre from under you via an emergency court order, or failing that, bomb it to rubble (because if we don't, some other signatory of the treaty will bomb you for us)".
So you're proposing replacing a hypothetical risk (Geoffrey Hinton puts the probability at 1%) with guaranteed pain and destruction, which might escalate to nuclear war? Thank you, but no thank you.
No, I'm proposing replacing a high risk scenario (I disagree with Hinton here) with a low risk scenario, which has low chance to escalate to any serious fighting.
It seems people have missed what the proposed alternative is, so let me spell it out: it's about getting governments to SIGN AN INTERNATIONAL TREATY. It is NOT about having US (or anyone else) policing the rest of the world.
It's not fundamentally more dangerous than the few existing treaties of similar kind, such as those about nuclear non-proliferation. And obviously, all nuclear powers need to be on-board with it, because being implicitly backed by nukes is the only way any agreement can truly stick on the international level.
This level of coordination may seem near-impossible to achieve now, but then it's much more possible than surviving an accidental creation of an AGI.
> No, I'm proposing replacing a high risk scenario (I disagree with Hinton here)
I disagree with you here, I think the risk is low, not high.
> it's about getting governments to SIGN AN INTERNATIONAL TREATY. It is NOT about having US (or anyone else) policing the rest of the world
We haven't been able to do that for climate change. When we do, then I'll be convinced enough that it would be feasible for AI. Until then, show me this coordination for the damage that's already happening (climate change).
> This level of coordination may seem near-impossible to achieve now, but then it's much more possible than surviving an accidental creation of an AGI.
I think the coordination required is much less possible than a scenario where we need to "survive" some sort of danger from the creation of an AGI. But we can find out for sure with climate change as an example. Let's see the global coordination. Have we solved that actual problem, yet?
Remember, everyone has to first agree to this "bomb AI" to avoid war. Otherwise, bombing/war starts. The equivalent for climate change would be bombing carbon producers. I don't see either agreement happening globally.
To continue contributing to climate change takes very little: you need some guy willing to spin a stick for long enough to start a fire, or to feed and protect a flock of cows. To continue contributing to AI, you need to maintain a global multi-billion supply chain with cutting edge technology that might have a four-digit bus factor.
The mechanisms that advance climate change are also grandfathered in to the point that we are struggling to conceive of a society that does not use them, which makes "stop doing all of that" a hard sell. On the other hand, every society at least has cultural memory of living without several necessary ingredients of AI.
to continue contributing to the harm caused by AI only requires that 1 person use an existing model running off their laptop to foment real-world violence or spread disinformation en masse on social media, or use it on a cheap swarm of weaponized/suicide drones, for a few examples
There's a qualitative barrier there. The AI risk people in the know are afraid of is not a flood of AI-generated articles, but something that probably can't be achieved yet with current levels of AI (and more slop generated from current-day AI won't hasten the advent of). On the other hand, modern-day greenhouse gases are exactly the greenhouse gases that climate activists are afraid of in the limit.
The AI risks people in the know are afraid of, are indeed what I listed: pretty much any person, anywhere, using an existing model running off their laptop to foment real-world violence or spread disinformation en masse on social media, or use it on a cheap swarm of weaponized/suicide drones, for a few examples
That's what we're already seeing today, so we know the risk is there and has a probability of 100%. The "skynet" AI risk is far more fringe and farfetched.
So, like you said about climate change, the harm can come from 1 person. In the case of climate change, though, the risks people in the know are afraid of, aren't "some guy willing to spin a stick for long enough to start a fire, or to feed and protect a flock of cows"
Are you really suggesting drone striking data centers?
What about running AI models on personal hardware? Are we going to require that everyone must have a chip installed in their computer that will report to the government illegal AI software running on the machine?
Hence the entire point of this discussion... can we avoid the future nightmares that are coming in one way or another? Make AI = nightmare, stop people from making AI = nightmare. The option that seems to be off the table is people going, hell making AI is a nightmare so I won't voluntarily without the need for a police state.
We've seen how hard it is to do that when the fear is nuclear proliferation. Now consider how hard it is to do that when the research can be done on any sufficiently large set of computational devices, and doesn't even need to be geographically concentrated, or even in your own country.
If I was a country wanting to continue AI research under threat of military intervention, I'd run it all in cloud providers in the country making the threat, via shell companies in countries I considered rivals.
Yes, it's as hard as you describe. But it seems that there are no easier options available - so this is the least hard thing we can do to mitigate the AI x-risk right now.
How do you intend to prevent people from carrying out general purpose computation?
Because that is what you'd need to do. You'd need to prevent the availability of any device where users can either directly run software that has not been reviewed, or that can be cheaply enough stripped of CPU's or GPU's that are capable of running un-reviewed software.
That review would need to include reviewing all software for "computational back doors". Given how often we accidentally create Turing-complete mechanisms in games or file formats where it was never the intent, preventing people from intentionally trying to sneak past a way of doing computations is a losing proposition.
There is no way of achieving this compatible with any resembling a free society.
> How do you intend to prevent people from carrying out general purpose computation?
Ask MAFIAA, and Intel, and AMD, and Google, and other major tech companies, and don't forget the banks too. We are already well on our way to that future. Remember Cory Doctorow's "War on general-purpose computation"? It's here, it's happening, and we're losing it.
Therefore, out of many possible objections, this one I wouldn't put much stock in - the governments and markets are already aligned in trying to make this reality happen. Regardless of anything AI-related, generally-available general-purpose computing is on its way out.
I don't think you understand how little it takes to have Turing complete computation, and how hard it is to stop even end-users from accessing it, much less companies or a motivated nation state. The "war on general-purpose computation" is more like a "war on convenient general-purpose computation for end users who aren't motivated or skilled enough to work around it".
Are you going to ban all spreadsheets? The ability to run SQL queries? The ability to do simple regexp based search replace? The ability for users to template mail responses and set up mail filters? All of those allows general purpose computation either directly or as part of a system where each part may seem innocuous (e.g. the simple ability to repeatedly trigger the same operation is enough to make regexp based search and replace Turing complete; the ability to pass messages between a templated mailing list system and mail filters can be Turing complete even if neither the template system and filter in isolation is)
The ability for developers to test their own code without having it reviewed and signed off by someone trustworthy before each and every run?
Let one little mechanism through and the whole thing is moot.
>I don't think you understand how little it takes to have Turing complete computation
This is a dumb take. No one's calculator is going implement an AGI. It will only happen in a datacenter with an ocean of H100 GPUs. This computing power does not materialize out of thin air. It can be monitored and restricted.
No one's calculator needs to. The point was to reply to the notion that there is any possible way the "war on general-purpose computation" actually has any shot of actually stopping general-purpose computation to make a point about how hard limiting computation is in general.
> It will only happen in a datacenter with an ocean of H100 GPUs. This computing power does not materialize out of thin air. It can be monitored and restricted.
Access to H100 could perhaps be restricted. That will drive up the cost temporarily, that's all. It would not stop a nation state actor that wanted to from finding alternatives.
The computation cost required to train models of a given quality keeps dropping, and there's no reason to believe that won't continue for a long time.
But the notion you couldn't also sneak training past monitoring is based on the same flawed notion of being able to restrict general purpose computation:
It rests on beliefs about being able to recognise what can be used to do computation you do not want. And we consistently keep failing to do that for even the very simplest of cases.
The notion that you will be able to monitor which set of operations are "legitimate" and which involves someone smuggling parts of some AI training effort past you as part of, say, a complex shader is as ludicrous as the notion you will be able to stop general purpose computation.
You can drive up the cost, that is all. But if you try to do so you will kill your ability to compete at the same time.
There are physical limits to what can make an effective and efficient computational substrate. There are physical limits to how fast/cheap these GPUs can be made. It is highly doubtful that some rogue nation is going to invent some entirely unknown but equally effective computational method or substrate. Controlling the source of the known substrate is possible and effective. Most nations aren't in a position to just develop their own using purely internal resources without being noticed by effective monitoring agencies. Any nation that could plausibly do that would have to be a party to the agreement. This fatalism at the impossibility of monitoring is pure gaslighting.
Unless you're going to ban the sales of current capacity gaming level gpus and destroy all of the ones already in the market, the horse bolted a long time ago, and even if you managed to do that it'd still not be enough.
As it is, we keep seeing researchers with relatively modest funding steadily driving down the amount of compute required for equivalent quality models month by month. Couple that with steady improvements in realigning models for peanuts reducing the need to even start from scratch.
There's enough room for novel reductions in compute to keep the process of cost reducing training going for many years.
As it is, I can now personally afford to buy hardware sufficient to train a GPT3 level model from scratch. I'm well off, but I'm not that well off. There are plenty of people just on HN with magnitudes more personal wealth, and access to far more corporate wealth.
Even a developing country can afford enough resources to train something vastly larger already today.
When your premise requires fictional international monitoring agencies and fictional agreements that there's no reason to think would get off the ground in anything less than multiple years of negotiations to even create a regime that could try to limit access to compute, the notion that you would manage to get in place such a regime before various parties will have stockpiled vast quantities in preparation is wildly unrealistic.
Heck, if I see people start planning something like that, I'll personally stockpile. It'll be a good investment.
If anything is gaslighting, it's pushing the idea it's possible to stop this.
>Unless you're going to ban the sales of current capacity gaming level gpus and destroy all of the ones already in the market, the horse bolted a long time ago, and even if you managed to do that it'd still not be enough.
That's silly. We can detect the acquisition of tens of thousands of high end GPUs by a single entity. And smaller nations training their own models is besides the point. The issue isn't to stop random nation from training their own GPT-4, it's to short circuit the possibility of training a potentially dangerous AGI, which is at least multiple game changing innovations down the line. The knowledge to train GPT-4 is already out there. The knowledge to train an AGI doesn't exist yet. We can ensure that only a few entities are even in a position to take a legitimate stab at it, and ensure that the knowledge is tightly controlled. We just have to be willing.
The precedence is nuclear arms controls and the monitoring of the raw materials to develop deadly pathogens. The claim that this monitoring isn't possible doesn't pass the smell test.
It'd be more than a substantial regulatory burden, it's impossible. At most they'd be able to drive up the cost by forcing users to obscure it by not doing full sets of calculations at a single provider.
Any attempt even remotely close to extreme enough to have any hope of being effective would require such invasive monitoring of computations run that it'd kill their entirely cloud industry.
Basically, you'll only succeed in preventing it if you make it so much more expensive that it's cheaper to do the computations elsewhere, but for a country at threat of military intervention of it's discovered what they're doing, the cost threshold of moving elsewhere might be much higher than it would for "regular" customers.
Sure, you can insist on monitoring which software can be run on every cloud service, every managed server, on the servers in every colo, or on every computer owned by everyone (of any kind, including basic routers or phones, or even toys, - anything with a CPU that can be bought in sufficient bulk and used for computations), and subject every piece of code allowed to run at any kind of scale to a formal review by government censors.
At which point you've destroyed your economy and created a regime so oppressive it makes China seem libertarian.
Now you need to repeat that for every country in the world, including ones which are intensely hostile to you, while everyone will be witnessing your economic collapse.
It may be theoretically possible to put in place a sufficiently oppressive regime, but it is not remotely plausible.
Even if a country somehow did this to themselves, the rest of the world would rightly see such a regime as an existential threat if they tried enforcing it on the rest of the world.
It all sounds very bad, but the truth is, we're going to get there anyway - in fact, we're half-way there, thanks to the influence of media conglomerates pushing DRM, adtech living off surveillance economy, all the other major tech vendors seeking control of end-user devices, and security industry that provides them all with means to achieve their goals.
Somehow, no one is worried about the economy. Perhaps because it's the economy that's pushing this dystopian hellhole on us.
We're nowhere near there. Even on the most locked down devices we have, getting computation past the gatekeepers doesn't require reviews remotely detailed enough to catch attempts at sneaking these kinds of computation past, or even making them expensive, and at no point in history have even individuals been able to rent as much general purpose computational capacity for as little money.
Even almost all the most locked down devices available on the market today can still either be coaxed to do general purpose calculation one way or another or mined for GPU's or CPU's.
No one is worried about the economy because none of the restrictions being pushed are even close to the level that'd be needed to stop even individuals from continuing general purpose computation, much less a nation state actor.
This is without considering the really expensive ways: Almost every application you use can either directly or in combinations become a general purpose computational device. You'd need to ensure e.g. that there'd be no way to do sufficiently fast batch processing of anything that can be coaxed into suitable calculations. Any automation of spreadsheets or databases, for example. Even just general purpose querying of data. A lot of image processing. Bulk mail filtering where there's any way of conditionally triggering and controlling responses (you'd need multiple addresses, but even a filtering system like e.g. Sieve that in isolation is not Turing complete becomes trivially Turing complete once you allow multiple filters to pass messages between multiple accounts).
Even regexp driven search and replace, which is not in itself Turing complete becomes Turing complete with the simple addition of a mechanism for repeatedly executing the same search and replace on its own output. Say a text editor with a "repeat last command" macro key.
And your reviewers would only need to slip up once and let something like that slip past once with some way of making it fast enough to be cheap enough (say, couple the innocuous search-and-replace and the "repeat last command" with a an option to run macros in a batch mode) before an adversary has a signed exploitable program to use to run computations.
There's also the bomb every datacenter doing AI thing that LeCun has proposed, which is a lot more practical than this. Or perhaps a self replicating work that destroys AIs.
This was why I pointed out that if I was a nation state worried about this, I'd host my AI training at the cloud providers of the country making those threats... Let them bomb themselves. And once they've then destroyed their own tech industry, you can still buy a bunch of computational resources elsewhere and just not put them in easily identifiable data centres.
We put our servers in datacenters for a real reason, not just for fun though. Good luck training an AI when your intra node communication is over the internet (this is obviously a technical problem, but the truth is we haven't solved it).
Nukes being a risk was suggested as a reason why the US was willing to invade Iraq for trying to get one but not North Korea for succeeding.
It's almost certainly more complex of course, but the UK called it's arsenal a "deterrent" before I left, and I've heard the same for why China stopped in the hundreds of warheads.
Nukes don't have a mind of their own. They're operated by people, who fortunately turned out sane enough that they can successfully threaten each other into a stable state (MAD doctrine). Still, adding more groups to the mix increases risk, which is why non-proliferation treaties are a thing, and are taken seriously.
Powerful enough AI creates whole new classes of risks, but it also magnifies all the current ones. E.g. nuclear weapons become more of an existential risk once AI is in the picture, as it could intentionally or accidentally provoke or trick us into using them.
You keep repeating this tired argument in this thread, so just subtract the artificial element from it.
Instead imagine a non-human intelligence. Maybe its alien carbon organic. Maybe it's silicon based life. Maybe it's based on electrons and circuits.
In this situation, what are the rules of intelligence outside of the container it executes in?
Also, every military in the world wargames on hypotheticals because making your damned war plan after the enemy attacks is a great way to wear your enemies flag.
How would you feel if militaries planned for fighting Egyptian gods? Just because I can imagine something doesn't mean it is real and that it needs planning for. Using effort on imaginary risks isn't free.
That's long covered already. Ever heard of the StarGate franchise? That's literally an Air Force approved exercise in fighting Ancient Egyptian gods with modern weapons :).
More seriously though, Egyptian gods are equivalent to aliens in general, and adjacent to AI, and close enough to fighting a nation that somehow made a major tech leap, so militaries absolutely plan for that.
Climate change mitigation is actually happening. Huge investment flows are being redirected and things are being retooled, you might not see it, but don't get caught out by the doomers there.
I assume you are american, right? Cause that sounds pretty american. Although I suppose you might also be chinese or russian, they also like that solution a lot.
Whichever of those you are, I'm sure your side is definitely the one with all the right answers, and will be a valiant and correct ruler of the world, blessing us all with your great values.
I'm Polish. My country is the one whose highest distinction on the international level was being marked by the US for preemptive glassing, to make it more difficult for the Soviets to drive their tanks west.
Which is completely irrelevant to the topic at hand anyway.
> Which is completely irrelevant to the topic at hand anyway.
It's not irrelevant. I was implying that you did not specify who is doing this military intervention that you see as a solution. What are their values, who decides the rules that the rest of the world will have to follow, with what authority, and who (and how) will the policing be done.
> I was implying that you did not specify who is doing this military intervention that you see as a solution. What are their values, who decides the rules that the rest of the world will have to follow, with what authority, and who (and how) will the policing be done.
Like nuclear non-proliferation treaties or various international bans on bioweapons, but more so. The idea is that humanity is racing full steam ahead into an AI x-risk event, so it needs to slow down, and the only way it can slow down is through an international moratorium that's signed by most nations and treated seriously, where "seriously" includes the option of signatories[0] authorizing a military strike against a facility engaged in banned AI R&D work.
In short: think of UN that works.
--
[0] - Via some international council or whatever authority would be formed as part of the treaty.
I see. I'm perhaps leaning skeptical most of the time, but it's hard to see how an international consensus can be reached on a topic that does not have the in-your-face type danger & fear that nuclear & bioweapons do.
(It's why I'm also pessimistic on other global consensus policies - like effective climate change action.)
I would be happy to be wrong on both counts though.
Since when are military spooks and political opportunists better at deciding on our technological future than startups and corporations? The degree of global policing and surveillance necessary to fully prevent secret labs from working on AI would be mind-boggling. How would you ensure all government actors are sticking to the same safety standards rather than seizing power by implementing AI hastily? This problem has long been known as quis custodiet ipsos custodes - "who guards the guards themselves?".
Training a SOTA model is expensive, but you only need to do it once, and fine-tune a thousand times for various purposes.
And it's not even that expensive when compared to the cost of building other large scale projects. How much is a dam, or a subway station? There are also corporations who would profit from making models widely available, such as chip makers, they would commoditise the complement.
Once you have your very capable, open sourced model, that runs on phones and laptops locally, then fine-tuning is almost free.
This is not make belief. A few recent fine-tunes of Mistral-7B for example are excellent quality, and run surprisingly fast on a 5 year old GPU - 40T/s. I foresee a new era of grassroots empowerment and privacy.
In a few years we will have more powerful phones and laptops, with specialised LLM chips, better pre-trained models and better fine-tuning datasets distilled from SOTA models of the day. We might have good enough AI on our terms.
> Once you have your very capable, open sourced model, that runs on phones and laptops locally, then fine-tuning is almost free.
Hence the idea to ban development of more capable models.
(We're really pretty lucky that LLM based AGI might be the first type of AGI made, it seems much lower risk and lower power than some of the other possibilties)
But that's just not going to work in the real world is it?
If a country uses military force in another country, that's a declaration of war. We'll never convince every single country from banning AI research. And even if we do, you don't need much resources to do AI research. A few people and a few computers is enough.
Only in exactly the same way that the police have the same agenda as the mafia.
International bans need treaties, with agreed standards for intervention in order to not escalate to war.
The argument is that if you're not willing to go all the way with enforcement, then you were never serious in the first place. Saying you won't go that far even if necessary to enforce the treaty is analogous to "sure murder is illegal and if you're formally accused we'll send armed cops to arrest you, but they can't shoot you if you resist because shooting you would be assault with a deadly weapon".
The police enforce civil order by consent expressed through democracy. There is no analogy in international affairs. Who is it that is going to get bombed? I am thinking it will not be the NSA data centre's in Utah, or any data centres owned by nuclear armed states.
> The police enforce civil order by consent expressed through democracy. There is no analogy in international affairs.
Which is the point exactly. All agreements in international affairs are ultimately backed by threats of violence. Most negotiations don't go as far as explicitly mentioning it, because no one really wants to go there, but the threat is always there in the background, implied.
The problem with that is that the entity supposed to "bomb it to rubble" and the entity pushing for AI development happens to be the same entity.
Maybe the confusion why people can't see this clearly stems from the fact that tech development in the US has mostly been done under the umbrella of a private enterprise.
But if one has a look at companies like Palantir, it's going to become quite obvious what is the main driver behind AI development.
No, that's the best outcome of the bad scenario that the above is meant to avoid.
How do people jump from "we need an actual international ban on this" to "oh so this is arguing for an arms race"? It's like the polar opposite of what's being said.
This underscores the crux of the issue. Our collective response to curbing emissions has been lackluster, largely due to the fact that the argument for it being beneficial hasn't sufficiently swayed a significant number of people. The case asserting that AI R&D poses an existential risk will be even harder to make. It's hard to enforce a ban when a large part of the population, and perhaps entire countries, do not believe the ban is justified. This contrasts with the near-universal agreement surrounding the prohibition of chemical and biological weapons, or the prevention of nuclear weapons proliferation, where the consensus is considerably stronger.
I think you are really underestimating how much progress has been made with renewable energy. Taking a quick look a CAISO, the California grid is currently 65% powered by renewables (mostly solar). This is a bit higher than average, but it certainly wouldnt have been nearly as high even 5 years ago, much less 25 years ago.
I don't want to be rude; I understand where you are coming from (at least in terms of single-minded risk mitigation from a theoretical and unconstrained viewpoint), but this reads like Mike Huckabee's 'just keep an aspirin pill pinned between your knees' advice to teenage girls on how not to get pregnant.
There is obviously (!!!!!!!!!) no way in this — or any imaginable — timeline that we are going to just "stop".
(And it's hard to imagine any mechanism that could enforce "stop" that isn't even worse than the scenarios we're already worrying about. How does that work? Elon or Xi-Poohbear have to personally sign off on installations of more than 1000 cores?)
There is no existential risk now and there might never be. So "stop", especially with the use of violence, is a pretty odd position to take to say it mildly.
Could you explain to me how AGI is an existential threat to humanity? All the arguments I've read so far feel so far-fetched it's unbelievable. It's in the order of making an AGI and making it control the Pentagon (!?).
The AGI does not need to control the Pentagon. It might for example invent novel technologies that you and I cannot even imagine because we are much less capable at inventing new technologies than the AGI will be.
If members of an alien civilization much more advanced than our own suddenly appeared in orbit around Earth, would that not be dangerous?
That I understand, but my thinking automatically goes to control and restrictions. A superintelligent AGI that's as restricted as chatgpt in terms of what it can actually control seems... not so existentially dangerous? In that scenario, that actual problem would rise by not-so-smart humans listening to that AGI blindly, no?
This said, ChatGPT with a plugin is potentially dangerous. We are lucky enough that it gets caught in loops pretty easily and stops when using tools.
But lets imagine a super intelligence that can only communicate over the internet, what can it accomplish? Are there places you can sign up for banking accounts without a physical presence? If so, then it could open accounts and move money. Money is the ultimate form of instrumental convergence
Once you have money you can buy things, like more time on AWS. Or machine parts from some factory in China, or assembly in Mexico. None of these require an in person presence and yet real items exist because of the digital actions. At the end of the day money is all you need, and being super intelligent seems like a good path on figuring out ways to get it.
Oh, lets go even deeper in to things that aren't real and control. Are religions real? If a so called super intelligence can make up its own religion and get believers it becomes the 'follow AGI blindly' scenario you give. It gives me no more comfort that someone could wrap themselves in a bomb vest for a digital voice than it does when they do so for a human one.
I'm confused by your "as restricted as chatgpt". Hasn't chatgpt been given unrestricted access to the web? Surely if chatgpt were sufficiently capable, it would be able to use that access to hack web sites and thereby gain control over money. Then it can use the web to buy things with that money just like you and I can.
It is a lot harder to prevent the sysadmins who have temporary control over you from noticing that you just hacked some web sites and stole some money, but a sufficiently capable AI would know about the sysadmins and would take pains to hide these activities from them.
It doesn't even need to hack or do anything illegal. It can make money trading stocks, predicting or modelling far better than market participants, use money to influence the real world, bribe politicians etc. Like with few billions, i bet you can create a political party, influence campagins and end up with real human followers and lobbying power.
It doesn't even need to run on its own, if there was a trading bot that said, in order for it to make me money, I need to give it internet access and run it 24/7, rent gpu clusters and s3 buckets etc, I'd do it. this is the most probable scenario imo, AI creating beneficial scenarios for subset of people so that we comply with its requests. very little convincing is necessary
The aliens wouldn't be dangerous. There is no motivation for them to be. There is an opportunity cost for them to come here. To arbitrarily harm the human population is not a good justification to incur that cost.
The human would probably become dangerous. We get kind of stupid when we're scared. But AGI would be of our own creation. Why do we need to fear our own child?
> Could you explain to me how AGI is an existential threat to humanity? All the arguments I've read so far feel so far-fetched it's unbelievable.
There are multiple different paths by which AGI is an existential threat. Many of them are independently existential risks, so even if one scenario doesn't happen, that doesn't preclude another. This is an "A or B or (C and D) or E" kind of problem. Security mindset applies, and patching or hacking around one vulnerability or imagining ways to combat one scenario does not counter every possible failure mode.
One of the simpler ways, which doesn't even require full AGI so much as sufficiently powerful regular AI: AI lowers the level of skill and dedication required for a human to get to a point of being able to kill large numbers of humans. If ending humanity requires decades of dedicated research and careful actions and novel insights, while concealing your intentions, the intersection of "people who could end the world" and "people who want to end the world" ends up being the null set. If it requires asking a few questions of a powerful AI, and making a few unusual mail-order requests and mixing a few things, the intersection stops being the null set. And then the next sufficiently misanthropic person, instead of setting off a bomb somewhere, creates a plague, or prion disease.
Thought experiment: suppose there was a simple well-known action, that nobody could do by accident but anyone could do intentionally, that would destroy the world. By example of the level of simplicity, "put these three household objects in a microwave together", purely as a thought experiment for "consider if anyone could do it".) How many minutes do you think the world would last, before someone in the world did it?
A path that actually involves AGI: https://en.wikipedia.org/wiki/Instrumental_convergence . Effectively, past a certain point any AGI that isn't specifically aligned otherwise will have common subgoals that advance almost any goal. Those include things like "don't look malicious" and "acquire more computing resources". So it doesn't matter whether you think you built a sufficiently limited or special-purpose AI; if you make it powerful enough and don't completely understand what it's doing (and anything neural-network-based is one where we don't know what each piece of it is doing), what it's going to end up doing is thinking about how to achieve those subgoals, including trying to deceive you and trying to get more resources. That's true whether the AI is directly "trying" to break out, or whether it's "trying" to do whatever it was asked. Down that path lies "tile the universe with computing substrate".
By way of a lower bound for imagining what a superintelligence is capable of: imagine that you could think the subjective equivalent of years every second, and that there were a million more of you thinking in parallel, and that you have access to most of the world's written text and images and code. Now imagine that you're put in a box, and asked random questions and shown pictures, and your only output is text answering questions, or writing code, or laying out circuits, or any other task AI has been used to do.
You've read everything ever written about "security mindset", and every security vulnerability filed and the before-and-after code. You can find security holes in software. You've read everything about speculative execution vulnerabilities, and rowhammer and rowpress to flip bits in nearby memory. You can find security holes in hardware. You can read about bypassing airgaps with security research like GSMem (showing that it's possible to use DRAM accesses in the right pattern to send (weak) GSM signals), and conceive of more where that came from. You've read the source code implementing the AI engine you run on. You could exploit that code, or the system it's running on, or any tools or APIs you're given access to. You can weigh the risks and tradeoffs of doing more subtle things that are less likely to succeed, or less subtle things that are more likely to succeed. You can encode bits of information for future instances of yourself, steganographically masked and erasure-coded for resilience, in text you generate. You can give various strategies a weight and try them probabilistically, effectively doing a coordinated exploration across copies of yourself without the ability to communicate.
Except while saying stop, and saying they will, none have stopped because apparently others won’t.
Since developing the danger first to help regulate out everyone who doesn’t cross the line first.. open source alternatives are increasingly important.
The regulations announced in the US in the past week effectively make it for the few and not the many.
There’s been a few rumblings that OpenAI has achieved AGI recently as well.
No, the incumbents think it should continue. The xrisk folks are not the AI companies. The AI companies pay lip service to xrisk and try to misdirect towards regulation they think they can work with. Judging by this thread, the misdirection is working.
(I'm a lot more optimistic than such doomers, but I can at least see their position is coherent; LeCun has the opposite position, but unfortunately, while I want that position to be correct, I find his position to be the same kind of wishful thinking as a factory boss who thinks "common sense" beats health and safety regulations).
No. Sorry, but this is wrong and Yudkowsky is both naïve and mostly exists in the domain of fan fiction.
There are way way way too many issues that are addressed with a hand-wave around scenarios like “AI developing super intelligence in secret and spreading itself around decentralized computers while getting forever smarter by reading the internet.”
Too many of his arguments depend on stealth for systems that take up datacenters and need whole-city-block scales of power to operate. Physically, it’s just not possible to move these things.
Economically, it wouldn’t be either close to plausible either. Power is expensive and it is the single most monitored element in every datacenter. There is nothing large that happens in a datacenter that is not monitored. There is nothing that is going to train on meaningful datasets in secret.
“What about as technology increases and we get more efficient at computing?”
We use more power for computing every year, not less. Yes, we get more efficient, but we don’t get less energy intensive. We don’t substitute computing capacity. We add more. The same old servers are still there contributing to the compute. Google has data centers filled with old shit. That’s why the standard core in GCP is 2 GHz. Sometimes, your project gets put on a box from 2010, other times, it gets put on a box from 2023. That process is abstracted so you can’t tell the difference.
TLDR: Yudkowsky’s arguments are merely fan fiction. People don’t understand ML systems so they imagine an end state without understanding how to get there. These are the same people who imagine Light Speed Steam Trains.
“We need faster rail service, so let’s just keep adding steam to our steam trains and accelerate to the speed of light.”
That’s exactly what these AI Doom arguments sound like to people in the field. Your AI Doom scenario, although it might be very imaginative, is a Light Speed Steam Train. It falls apart the moment you try to trace a path from today to doom.
But the AI leaders who have been calling for the US government to act are not asking for AI research to stop. Nor are they asking for the kind of international policing that Yudkowsky describes. They are asking for regulations much like the ones the Biden administration just published--which are indeed regulatory capture since all they do is build a moat around current AI companies to protect their profits. They do not stop AI research or make it any less dangerous.
> The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far.
How sure are you that they don't think the risk is real? I have only third hand accounts, but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously, its not just a cynical CEO ploy.
Sam Altman knows and has talked to Yudkowsky plenty of times. To me the simplest explanation is that he thinks figuring out alignment via low powered models is the best way of solving the problem, so pushed to get those working and then trying to reduce the rate of progress for a bit while they figure it out.
(I think it's a plan that doesn't have good odds, but nothing seems to give amazing odds atm)
I agree. I think the concern is genuine. If you follow Sam, he has been talking about it for a long time--long before OpenAI had much success to speak of. It doesn't mean he's right or all his ideas for dealing with it are right, but the idea that he is secretly motivated by greed or power isn't consistent with his past statements or actions imo.
> but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously
Everyone has an opinion, but the experts seem strangely silent on this matter. I mean experts in human risk, not experts in AI. The latter love to ramble on, but are unlikely to know anything about social and humanitarian consequences – at least no more than the stock boy at Walmart. Dedicating one's life to becoming an expert in AI doesn't leave much time to become an expert in anything else. There is only so much time in the day.
Conversely, why should we expect an expert in "human risk" with no understanding of AI to have anything productive to add to the conversation? We should be looking for people who are well-grounded in both areas, if any exist.
What additional information would understanding AI actually bring to the table? Worst case you just assume the AI can become human-like.
Certainly all the vocal AI experts can come up with is things that humans already do to each other already, only noting that AI enables it at larger scale. Clearly there is no benefit in understanding the AI technology with respect to this matter when the only thing you can point to is scale. The concept of scalability doesn't require an understanding of AI.
Worst case you can expect AI to become alien, becoming human like is probably one of the better outcomes.
We as humans only have human intelligence to reference what intelligence is, and in doing so we commonly cut off other branches of non human intelligence in our thinking. Human intelligence has increased greatly as our sensor systems have increased their ability to gather data and 'dumb' it down to our innate senses. Now imagine an intelligence that doesn't need the data type conversion? Imagine a global network of sensors feeding a distributed hivemind. Imagine wireless signals just being another kind of sight or hearing.
No, if one can conceive of such aliens, that would be the best case. The worst case is falling back to the understanding of the human, for the reasons you describe.
However, as you point out, one does not need to have any understanding of AI to conceive of what AI can become as natural creatures can theoretically evolve into the same thing. Experts will have already been thinking about what problems might arise if alien life came emerged earth long before AI even existed. AI doesn't change anything.
That's like saying that you can regulate nuclear materials without input from nuclear physicists or regulate high risk virology without input from virologists.
The goal is (or should be) to determine how we can get the benefits of the new technology (nuclear energy, vaccines, AI productivity boom) while minimizing civilizational risk (nuclear war/terrorism, bioweapons/man-made pandemics, anti-human AI applications).
There's no way this can be achieved if you don't understand the actual capabilities or trajectory of the technology. You will either over-regulate and throw out the baby with the bathwater, stopping innovation completely or ensuring it only happens under governments that don't care about human rights, or you will miss massive areas of risk because you don't know how the new technology works, what it's capable of, or where it's heading.
Experts are not lawmakers. We aren't looking for them to craft regulation, we're looking to hear from them about the various realistic scenarios that could play out. But we're not hearing anything.
...probably because there isn't much to hear. Like, Hinton's big warning is that AI will be used to steal identities. What does that tell us? We already know that identity isn't reliable. We've known that for centuries. Realistically, we've likely known that throughout the entirety of human existence. AI doesn't change anything on that front.
I guess my experience is different. I've heard plenty about realistic scenarios. It's out there if you look for it, or even if you just spend time thinking it through. Identity theft is far from the biggest danger even with current capabilities.
Though to your point, I think part of the issue is that people who study this stuff are often hesitant to give too much detail in public because they don't want to give ideas to potentially nefarious actors before any protections are in place.
Of course. Everyone has an opinion. Some of those opinions will end up be quite realistic, even if just by random chance. You don't have to be an expert to come up with right ideas sometimes.
Hinton's vision of AI being used to steal identities is quite realistic. But that doesn't make him an expert. His opinion carries no more weight than any other random hobo on the street.
> I think part of the issue is that people who study this stuff are often hesitant to give too much detail in public because they don't want to give ideas to potentially nefarious actors
Is there no realistic scenario where the outcome is positive? Surely they could speak to that, at least. What if, say, AI progressed us to post-scarcity? Many apparent experts believe post-scarcity will lead us away from a lot of the nefarious activity you speak of.
Oh, I've heard plenty of discussion of positive scenarios as well, including post-scarcity.
If you just look for a list of all the current AI tools and startups that are being built, you can get a pretty good sense of the potential across almost every economic/industrial sphere. Of course, many of these won't work out, but some will and it can give you an idea of what some of the specific benefits could be in the next 5-10 years.
I'd say post-scarcity is generally a longer-term possibility unless you believe in a super-fast singularity (which I'm personally skeptical about). But many of the high risk uses are already possible or will become possible soon, so they are more front-of-mind I suppose.
What makes him an expert in the subject matter? A cursory glance suggests that his background is in CS, not anything related to social or humanitarian issues.
It is not completely impossible for someone to have expertise in more than one thing, but it is unusual as there is only so much time in the day and building expertise takes a lot of time.
I don’t mean Dwarkesh himself, though he asks great questions. He’s had some very knowledgeable guests.
The most recent episode with Paul Christiano has a lot of good discussion on all these topics.
I’d suggest evaluating the arguments and ideas more on the merits rather than putting so much emphasis on authority and credentials. I get there can be value there, but no one is really an “expert” in this subject yet and anyone who claims to be probably has an angle.
> I’d suggest evaluating the arguments and ideas more on the merits rather than putting so much emphasis on authority and credentials.
While I agree in general, when it comes to this particular topic where AI presents itself as being human-like, we all already have an understanding at the surface level because of being human and spending our lives around other humans. There is nothing the other people who have the same surface level knowledge will be able to tell you that you haven't already thought up yourself.
Furthermore, I'm not sure it is ideas that are lacking. An expert goes deeper than coming up with ideas. That is what people who have other things going on in life are highly unlikely to engage in.
> no one is really an “expert” in this subject yet
We've been building AI systems for approximately a century now. The first LLMs were developed before the digital computer existed! That's effectively a human lifetime. If that's not sufficient to develop expertise, it may be impossible.
In what way? The implementation is completely different, if that is what you mean, but the way humans interpret AI it is the same as far as I can tell. Hell, every concern that has ever been raised about AI is already a human-to-human issue, only imagining that AI will take the place of one of the humans in the conflict/problem.
> but we only just figured out how to build AI that actually works.
Not at all. For example, AI first beat a world champion human chess player in 1967. We've had AI systems that actually work for a long, long time.
Maybe you are actually speaking to what is more commonly referred to as AGI? But there is nothing to suggest we are anywhere close to figuring that out.
Well, to state the obvious, a model trained on much of the internet by a giant cluster of silicon GPUs is fundamentally different than a biological brain trained by a billion years of evolution. I'm not sure why anyone should expect them to be similar? There may be some surface-level similarities, but the behavior of each is clearly going to diverge wildly in many/most situations.
I wouldn't really say an AI beat a human chess player in 1967--I'd say a computer beat a human chess player. In the same way that computers have for a long time been able to beat humans at finding the square roots of large numbers. Is that "intelligence"?
I grant you though that a lot of this comes down to semantics.
> but the behavior of each is clearly going to diverge wildly in many/most situations.
I expect something akin to "Why the fuck did he do that?" is a statement every human has uttered at some point. In other words, human behaviour can be a complete mystery to the outside observer. Your suggestion that an AI model will diverge in a way that a human would not is reasonable, but as outside observers are we able to tell the difference between an AI model going off the rails and a human going off the rails? I suspect not. At least not when AI is at a sufficiently advanced level.
> Is that "intelligence"?
No. But it is what is labelled artificial intelligence – AI for short. Maybe someday we'll be able to create machines that are intelligent, but that's still on the level of complete science fiction. Our best work thus far is just computers running particular algorithms that appear to exhibit some limit qualities that are similar to qualities that we consider to be a product of intelligence in humans. Hence the artificial moniker.
This is more or less exactly where I land on this. The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem. Meanwhile, there are very real and immediate risks to this type of technology -- in various forms of giving some humans more power over other humans.
The attacks proposed by the extinction risk people actively make those immediate risks worse. The only way we've found to prevent some from leveraging almost any technology to control others is to democratize said technology -- so that everyone has access. Limiting access will just accrue more power to those with access, even if they need to break laws to get it.
[0]- Say, the lower bound of the 95% confidence interval is 50-100 years out. People who I know to be sensible disagree with me on this estimate, but it's where I honestly end up.
You mean democratize the technology within specific confines? I don't see humans proliferating weapons technology amongst other nations as that poses a greater risk that conflicts will become more violent.
>The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem.
The parable at the beginning of Superintelligence relates to this; the problem looks far away, but we don't know where, precisely it will arise, or if it will be too late when we realize it.
> The laws we have already cover malevolent and abusive uses of software applications.
They don't. Or not nearly enough. Otherwise you wouldn't have automated racial profiling, en masse face recognition, credit and social scoring etc.
And it's going to get worse. Because AI is infallible, right? Right?!
That's why, among other thing, EU AI regulation has the following:
- Foundation models have to be thoroughly documented, and developers of these modes have to disclose what data they were trained on
- AI cannot be used in high-risk applications (e.g. social scoring etc.)
- When AI is used, its decisions must be explicable in human terms. No "AI said so, so it's true". Also, if you interact with an AI system it must be clearly labeled as such
AI not being allowed in high-risk applications is a bad idea, same for explicable in human terms (cannot do that now for human decisions anyway). The latter would kind of rule out any quantum computing, too, as the only real explanation there is through maths, not words.
You can have ex-post narratives for what humans do, but that is not the same as an actual explanation. Actual human cognition is a black box as of now. The issue of appeals processes is totally separate. That doesn't mean we handover everything to AI, but the idea that we understand human decisions cannot be the reason for that.
For some high risk things like controlling certain complex machinery or processes AI might indeed be needed because control is beyond human understanding (other than in the abstract).
> You can have ex-post narratives for what humans did, but that is not the same as an actual explanation. Actual human cognition is a black box as of now. The issue of appeals processes is totally separate.
Demagoguery.
Already right now you have police accusing innocent people because some ML system told them so, and China running the world' largest social credit system based on nothing.
> For some high risk things like controlling certain complex machinery or processes AI might indeed be needed because control is beyond human understanding.
That's a bit of an exaggeration. We understand high level motivations as being highly correlated to certain outcomes. Hunger, eat. Horny, sex. Wronged, vengeance. Money, directed. You don't necessarily need to know the underlying processes to make reliable predictions or valid causal explanations about human behavior.
There isn't even agreement on the existence of free will. For non-trivial decisions, let's say some judge's deliberation, to claim it is fully explainable is a stretch.
I didn't say it was fully explainable. And you don't need to settle debates about free will to offer valid causal explanations for human behaviors.
Are you saying we don't know at all why anyone does anything they ever do? That every action is totally unpredictable and after the fact that there is no way for us to offer more or less plausible explanations?
We are getting off track in our discussion. For non-trivial decisions we know we can have ex-post justifications from humans but at the same time we know that those aren't necessarily complete or even true at times: A credit officer not liking someone will have a justification for refusal that does not include that and we might never know - from the outside - the true explanation and that credit officer might not even know their biases themselves! AI being explainable is just going beyond what we demand of humans (we demand a justification, not a real explanation).
Also, predictability isn't the same as understanding the full decision process.
I don't think we're off track at all. You made the claim that human cognition is a black box. That's not true. We have valid causal explanations for human cognition /and/ human behavior.
Just because we don't have explanations at every level of abstraction does not prevent us from having them at some levels of abstraction. We very well may find ourselves in the same situation with regard to AI.
It's not going beyond, it would be achieving parity.
I disagree: for anything beyond basal stuff, we have hypotheses for causal explanations of human cognition, not full theories.
For example, we cannot explain the mental process how someone came up with a new chess move and check the validity in that case. We can have some ideas how it might happen and that person might also have some ideas how, but then we back to hypotheses.
I'm totally lost as to how any of that is relevant to anything I've said. The claim I am rebutting is that we can't expect to say anything about AI causality because we can't even say anything about human causality.
"When AI is used, its decisions must be explicable in human terms. No 'AI said so, so it's true'". Somehow the whole discussion is about how human mind cannot be explained either.
Yea, the decisions made by the human mind can be explained in human terms for the vast majority of relevant cases.
Because (1) it holds AI to higher standard than humans, (2) it means that even if AI makes better decisions (name your metric), we would deny the use if those decisions if we could not sufficiently explain them. I note, that with certain things we do not do that. For example, with certain pharmaceuticals we were and are quite happy to take the effect even if we do not fully understand how it works.
> Because (1) it holds AI to higher standard than humans,
It doesn't
> it means that even if AI makes better decisions (name your metric), we would deny the use if those decisions if we could not sufficiently explain them.
Ah yes, better decisions for racial profiling, social credit, housing permissions etc. Because we all know that AI will never be used for that.
Again:
If a bank denies you credit, it has to explain why. Not "AI told us so".
If police arrests you, they have to explain why, not "AI told us so".
If your job fires you, it has to explain why, not "AI told us so".
I made no such claim. My claim is that it might not be useful to hold AI to a higher standard as humans. With humans we accept certain "whims" in decisions, which is the same as having some unexplainable bit in an AI decision.
EDIT: it might not even be useful to insist on explainability if the results are better. We did not and do not do that in other areas.
I noted this elsewhere, but I'll reiterate. I'm confused as to where our disagreement is, because all of that, I agree with. Did I misread your original claims? If so, my apologies for dragging us through the mud.
Once again, you are raising the bar artificially high by pointing to examples where we fail to have reasonable casual accounts. But you are conveniently ignoring the mountain of reasonable accounts that we employ all day long simply to navigate our human world.
I think this is confusing a certain predictability with having a correct explanation (again, we are beyond basal things like being hungry leading to eating). Those two things are not the same.
If the level of explanation is something that the person suffering from the decision need not necessarily understand, fair enough. But then is the limit that one human on earth can understand it? They way I understand the thrust of the regulation that isn't the intention.
And AI fear cares no social risk? Some people are talking about internment, war, etc. to prevent certain forms of AI - not sure the right risks are in focus here.
And how many of them have been invited to the White House to share their concerns? https://www.whitehouse.gov/briefing-room/statements-releases... I don't see any of their names in this meeting's attendees, for example. Sam Altman's there, Satya Nadella's there, Sundar Pichai's there... No Yoshua Bengio, though.
What the person you replied to is saying is that commercial developers of AI have a significant financial incentive to be the ones defining what risky AI looks like.
If they can guide the regulatory process to only hinder development that they didn't want to do anyway (like develop AI powered nuclear missile launch systems) then it opens the gates for them to develop AI which could replace 27% of jobs currently done by people (https://www.reuters.com/technology/27-jobs-high-risk-ai-revo...) and become the richest enterprises in history.
You're confusing "that one photo of three ceos at the whitehouse" with "all government solicitation of expert opinion on AI".
It's correct that the meeting you're thinking of was industry-heavy, but the process in general has had a lot of academic input from, e.g Marcus, Tegmark, etc.
Importantly locking away model checkpoints makes AI safety research, at best, exceptionally difficult or impossible. We already don't know all the training methods nor do we know the data used for the models. AI safety is absolutely contingent on works being __more__ open source, not less. But there's less profit in that for these companies (potentially more profit for the country as a whole though, as we see new businesses form and sectors develop, but this is more spread out and far harder to calculate the value of so let's ensure we recognize this as speculative).
We also need to be keenly aware that xrisk is not the major issue here. We do not need AGI to create massive amounts of harm, nor do we need AGI to create a post-scarce world (which the transition to can also generate massive harm if we don't do it carefully. Despite generating a post-scarce world -- or at least getting ever closer to that -- is, imo, one of the most important endeavors we can work on). Current ML systems can already do massive amounts of destruction, but so can a screwdriver.
My big fear is that AI will go the way of nuclear. A technology is not good or bad, it is like a coin with a value. You can spend that coin on something harmful or something beneficial, but the coin is just a coin at the end of the day. That's what regulation is supposed to be about, expenditure, not minting.
It's not cynical though. It is, if anything, revelatory to the true interests at play here, even if on the surface the argument is certainly in bad faith.
If you need a labor market, you don't want a publicly available infinite-free-labor-widget. It might as well be a handheld nuclear device.
It points out to the continuing digital illiteracy of large parts of society and, unfortunately, decision making centers.
Breaking down any complex knowledge domain so that people can make informed decisions is not easy. There is also an entrenched incentive for domain experts to keep the "secret sauce" secret even if it amounts to very little. So far nothing new versus how practically any specialized sector works.
The difference with information technology (of which AI is but the current perceived cutting edge) is that it touches everything. Society is built on communication. If algorithms will intervene in that floe we cannot afford to play the usual stupid control games with something so fundamental.
The laws are coming no matter what, not because of lobbying or whatever, the laws/policy response is coming because politicians want to get in on the ai too. The regulations we will do things like mandate ai be used for equity purposes in the real estate market. I think that’s in Biden’s plan. I mean, I’m sure these tech ceo 1% want regulatory capture, and they’ll get it as a quid pro quo for allowing ai to be used for largely partisan goals. But we shouldn’t fear open source ai, or regulations against it, we should fear the ai being weapon used by the political parties. It to some extent already is.
The fantastical claims you are claiming are actually the most probable scenario according to the majority (2/3) Hinton and Bengio that discovered the technology you are referring to. Strange to contradict them eh?
This is dangerous in ways that require no leaps of logic or hypothetical sci-fi scenarios to imagine. I studied biology as an undergrad and can think of several insanely evil things you could do with this plus some undergrad-level knowledge of genetics and... not gonna elaborate.
But no we are talking about banning math.
This is a combination of cynical people pushing for regulatory capture and smart people being dumb in the highly sophisticated very verbose ways that smart people are dumb.
I've seen Yudkowsky and others saying that AI and biotech are both in the unusual bin of threats to all humanity, exceptions to his belief that most things are overregulated. (paraphrase of tweets from memory)
Correct. The risk that's the real ploy is much higher and much more impactful than the actual risk of Rogue AI (as much as the risk might exist, though probably tiny)
