I'm gonna request more explanations and proof, or at least theoretical path on using Expedia, Zapier, Instacart, Kayak etc. to dominate the world and kill every single human on earth.
Explanations, sure. My point was that yes, ChatGPT is indeed an entity which cannot interact with the world except through reading and writing text. This would be a lot more comforting if people were not rushing to build ways to turn its text output into actions in the physical world as fast as possible.
Imagine a mob boss whose spine was severed in an unfortunate mob-related accident. The mob boss cannot move his arms or legs, and can only communicate through speech. Said mob boss has it out for you. How worried are you? After all, this mob boss cannot do things. They create speech in response to prompts. And that's it!
I actually don't agree with Eliezer that the primary threat model is a single consequentialist agent recursively bootstrapping its way to uncontested godhood. But there is a related threat model, that of "better technology allows you to make bigger mistakes faster and more vigorously, and in the case of sufficiently powerful AGI, autonomously".
In terms of proof that it's possible to destroy the world and kill all humans, I will not provide that. No matter how poetic of an ending it would be for humanity if it ended because someone was wrong on the internet, and someone else felt the need to prove that they were wrong.
I don’t disagree with the “AI will upend the world so we have to prepare”, it’s the “AI will kill everyone” that I have issue with.
And your mob boss example is a good reason why: it doesn’t extrapolate that much. There is no case where a mob boss, or a disabled Hitler for that matter, can kill everyone and ends humanity.
The mob boss analogy breaks down when they need assistance from other humans to do stuff. To the extent that an AI can build its own supply chains, that doesn't apply here. That may or may not be a large extent, depending on how hard it is to bootstrap something which can operate independently of humans.
The extent to which it's possible for a very intelligent AI with limited starting resources to build up a supply chain which generates GPUs and enough power to run them, and disempower anyone who might stop it from doing so (not necessarily in that order), is a matter of some debate. The term to search for is "sharp left turn".
I am, again, pretty sure that's not the scenario we're going to see. Like at least 90% sure. It's still fewer 9s than I'd like (though I am not with Eliezer in the "a full nuclear exchange is preferable" camp).
I will take an example that Eliezer has used and explain why I think he is wrong: AlphaGo. Eliezer used it as an example where the AI just blew through humanity really quickly, and extrapolate it to how an AGI will do the same.
But here is the thing: AlphaGo and subsequent AI didn’t make the previous human knowledge wrong at all, most of what was figured out and taught are still correct. There are changes at the margin, but arguably the human are on track to discovered it anyway. There are corner sequences that are truly unusual, but the big picture of playing style and game idea are already on track to be similar.
And it matters because things like nanotech is hard. Building stuffs at scale is hard. Building factories at scale is hard. And just because there is a super intelligence being doesn’t mean they become a genie. Just imagine how much trouble we have with distributed computing, how would a cluster of computing gives rise to a singularity of an AI? And if the computer device has to be the human brain size, there is a high chance it hits the same limits as our brain.
I mean I think his point there was "there is plenty of room for systems to be far, far more capable than humans in at least some problem domains". But yeah, Eliezer's FOOM take does seem predicated on the bitter lesson[1] not holding.
To the extent I expect doom, I expect it'll look more like this[2].
Not endorsing the arguments either way but let's say DNA printing (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...) or something like Stuxnet or crashing a nuclear-power country's stock market or currency through trading while making trades appear to come from another country or by causing bank runs through hacking social media or something like WhatsApp or through deep fakes or by having human helpers do stuff for the AI voluntarily in order to get very rich...
There's a web plugin too. It can issue GET requests. That's enough to probe a lot of interesting things, and I'll bet there's an endpoint somewhere on the web that will eval any other web request, so now you've opened up every web accessible API - again, all theoretical, but at least not too far removed from an exploit.
