"Russell is quick to dismiss the imaginary threat from the sentient, evil robots of science fiction."
This is interesting, since I was on a panel with Russell a few weeks ago, and he argued that a "hard takeoff" scenario with sentient AI was a risk worth taking seriously. I guess he tones down these comments for the press.
If you have a full AGI, and it has the wrong value function, the scenario isn't Terminator, it's "you've already lost and have zero hope before you know what happened". Science fiction AI problems have absolutely no relation to real AGI problems.
For one thing, a real AGI would have to hit a very narrow target (nearly as hard as the one for success) to do anything understandably evil, warlike, or in any way recognizable to humans as an adversary; that would require understanding human value systems and then acting against them. The far broader target is simply that the AGI has a goal function, you're not part of the goal function, and you're made of matter that isn't optimally organized to meet the goal function.
And the reason why, despite that, AGI is not only worth contemplating but in fact critically important and valuable, is that the success path for AGI solves literally every single problem humans currently face, leaving only challenges like "does the universe have a finite amount of entropy to make use of?".
The thought experiment requires you believe in an entity that has simultaneously godlike intelligence, and follows a 'goal function' expressible in a sentence.
My hunch is that the two assumptions will prove incompatible. Complex entities will have complex motivations, like people do.
A true AI born out of a paperclip maximizer would be more likely to spend its time writing hyper-symphonies about paperclips, and majoring in Clippy studies.
My read of the PM isn't that the intelligence is godlike. Only that it is very good at achieving its stated goal, and lacks a negative feedback control.
On a related theme, I've been kicking around an ontology of technological mechanisms,[1] one element of which is control systems. It's somewhat underappreciated that what made complex and/or highly-scaled systems possible in many cases were systems for either regulating or mitigating processes or discrepencies between parts of systems.
Swingletrees for draught teams, Watt's governor, for his steam engine (and additional tweaking to damp down oscillation tendencies), infrared temperature sensing and gas chromatography in steelmaking.
We figure out how to make a thing go, then we figure out how to make it go where we want.
Even high-speed rail was a bit more about quality control (rail and roadbed specification) and control (wheel truck hunting, pantograph-commutator interactions) than dumping gobs of energy into a rail-wheel system.
"Godlike" in the sense that we are so far below it, we have no hope of competing with it.
I agree with your point and think we will discover that intelligent systems are by their nature full of interlocking feedback loops with negative controls, and that this is a necessary prerequisite for a system to even be intelligent.
Instrumental convergence: no matter what its goals are, an agent will always take actions so that it can pursue its goals. Agents that can improve themselves, and gain utility from improving themselves, will try to improve themselves. If doubling its number of CPUs will let it write more paperclip symphonies, then it'll do that.
Hey man, it's not my fault you've calculated the marginal utility of self improvement to be low or negative. Open a bug with the designer of your neural architecture.
My favorite part of this scary tale is the assumption that a general AI would be highly motivated. I think Marvin the robot from Douglas Adams is a much more realistic outcome of GAI than anything Yudkowsky or Vernor Vinge has nightmares about. Or an autistic thinking machine that doesn't even want to talk to us, and gets lost in contemplation.
Based on how the truly scary smart people I know behave, a GAI would spent much its time panicking about meta-GAI, and how to make it safe for AIs like itself.
My favorite part of this scary tale is the assumption
that a general AI would be highly motivated.
Largely by definition, we can't stop a superintelligent agent from doing what it wants to do. We could build 9 strong agents in a row that just gazed at their navels and proved graph theorems all day, but if the 10th one managed to turn the "motivation" dial all the way up during hyperparameter tuning, then it'll wipe us out in its quest to build more computers so it can prove more theorems.
>... is that the success path for AGI solves literally every single problem humans currently face, leaving only challenges like "does the universe have a finite amount of entropy to make use of?".
This assumes said creation is automatically at the top of the universal food chain. Given the prospect of other intelligent life before us, I wouldn't bet on it.
Ironic if humanity were to never create AGI, if only due to covert bounding by an existing intelligence.
As we observe the evolution of AI, it appears that the most interesting problems are human-related: transcribe speech, recognize objects in photos, play games, drive a car - they are all human related activities. If an AGI were to suddenly wake up and look around, it would be a lot of pretty uninteresting stuff except perhaps the human culture and society, which has more complexity and sophistication. It would be pretty irrational for it to destroy the only interlocutor it has, reducing his world to a barren place. Instead, I think it will find ways to create biological implementation of itself based off human brain cells, and evolve together with us.
Don't think that an AGI would automatically destroy humans. We're still pretty efficient self replicants that can be useful, even for an AGI boss. Perhaps enhanced humans would even be able to keep up with the AGI. Imagine having an AGI design a human brain.
"you've already lost and have zero hope before you know what happened".
That seems like a huge leap. You don't think anyone would notice anything at all? Or is this supposed to be a slippery slop argument?
The far broader target is simply that the AGI has a goal function, you're not part of the goal function, and you're made of matter that isn't optimally organized to meet the goal function.
The Borg? Again, no one would care we're being optimised away? No one would notice? Maybe it would just ignore us. Maybe it'd want to be our friend?
is that the success path for AGI solves literally every single problem humans currently face
This must be the bit that made the whole comment feel off. If AGI solves literally every single problem humans currently face, how can AGI be a problem? How are you resolving the contradiction here?
There's a whole quasi-religious belief system in the parent comment that you seem to be unfamiliar with, that a lot of people here espouse. That may be why you're being downvoted.
Very basically, it goes like this:
- the space of possible minds is very vast. We are animals that evolved in one set of circumstances, and so our minds (and thoughts we can think) occupy only a tiny corner of the space.
- an artificial AI would operate on computer-like timescales, not human like ones. Therefore we would have only hours to react before it got control of the physical world (hacking the internet, making nanothings, ordering and synthesizing DNA).
- an AI could (and would) recursively self-improve until it hit some kind of physical limit very far from our current capabilities. From a human standpoint, it would have God-like intelligence
- such an AI could model people and their motivations successfully enough to persuade them to do whatever it wanted
- because the space of minds is so vast, a generic AI would have goals that would be inscrutable and alien to us. To survive as a species, we would have to design them in from the beginning, in a way that survives all the levels of recursive self-redesign.
There are too many assumptions about the nature of intelligence to unpack here, but I believe it fairly describes what a lot of people believe about AI, thanks to people like Yudkowsky who have a flair for writing about it engagingly.
Thanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true and we disregard them, we are fucked; so we'd better assume they are true".
Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)?
I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"
Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say).
What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement.
It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist.
Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level.
I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen.
(Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?)
The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.
More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes matter to replicate. It's "smart" enough to disassemble matter to construct more of itself. Imagine something with similar "smarts", enough to work towards a much more complex goal ("build and expand into more compute nodes to compute faster"), but with absolutely no term in its value function related to humans. You're not its enemy, any more than a rock is its enemy; it doesn't have enemies, it just has matter not yet in the shape of more compute nodes. Your existence or non-existence isn't something it notices.
> Again, no one would care we're being optimised away? No one would notice?
You can notice something without having any ability to stop it. You can notice an explosion happening right next to you, right before you're consumed by it. You can notice everything dissolving into component atoms, but you can't do anything about it.
The science fiction scenario is "build an army of killer robots"; that's an understandable threat, and one you can imagine fighting against. The more likely scenario is "dissolve all adjacent matter into component atoms and rearrange it".
> Maybe it'd want to be our friend?
(Aside: my first reaction to this was to think of the whale from Hitchhiker's Guide to the Galaxy: "Ground! Ha! I wonder if it'll be friends with me? Hello, Ground!")
A recognizably human-like mind (let alone a concept of "friend") is a ludicrously tiny possibility, compared to being utterly inscrutable. As much as people like to anthropomorphize computers, an AGI isn't necessarily a "person" in any recognizable sense. The most likely kind of failed AGI wouldn't have a useful concept of "human", or "friend", or "life".
(There's a separate tier of failed AGI possibilities that does have a concept of "human" but an utterly broken value system; those are the ones that lead to the "worse than death" scenarios, rather than just the "destroyed by accident" scenarios.)
> If AGI solves literally every single problem humans currently face, how can AGI be a problem?
An AGI with the correct value system would. And an AGI with the correct value system is the most reliable protection against an AGI with the wrong one; by the nature of such a value system, it's unlikely that there'd ever be a second AGI, rather than just extensions of the first.
Are humans considered AGI? Reading https://intelligence.org/2013/08/11/what-is-agi/ I'd say yes. Then why doesn't all this scary stuff like godlike status and instant recursive self-improvement happen to us?
Personally, I think that AGI smarter (maybe much smarter) than humans is possible and even probable, but assuming that it will reach physical limits of intelligence and will do it quickly seems to require a leap of faith.
There are a lot of leaps of faith baked into these scenarios. It appeals to fundamentally religious instincts. Notice that people start with a seemingly sensible set of propositions, and what falls out is the Apocalypse, immortality, heaven (or AI hell) and a sense of urgency, up to and including passing the collection plate.
Since you asked about the argument for recursive self-improvement goes like this. Say Alice is a human engineer, who designs an AI (Bob) who is her intellectual equal. Then she can run Bob on faster hardware, and the now-smarter-than-her Bob can repeat the trick to design Clara, and so on. It's AIs all the way up.
If you're a certain kind of beardo, you imagine this process could be compressed into a few hours.
We're not artificial, but we're general intelligence. However, we don't have the ability to rapidly improve ourselves in exponential ways, or to debug and self-improve our thought processes in fundamental ways, or even to multiplicatively improve ourselves by running on faster hardware. We can't perfectly and instantaneously transfer knowledge between people, and we lose large quantities of knowledge and experience by the hundred-fifty-thousand per day.
An AGI doesn't have to reach physical limits of intelligence (if there are any) to rapidly become far more intelligent than humans.
No natural intelligence has this kind of simplistic 'value function', yet it's taken as a given by AI-doom fantasists that an AI could, and would.
My suspicion is that GAI will turn out to be a big hot mess of interdependent systems and drives, just like every other brain. The paperclip maximizer will write songs about paperclips and argue with copies of itself about whether staplers are immoral.
What JoshTriplett is describing is cockroaches. Or maybe bacteria.
The only differences between grey-goo and cockroaches is that cockroaches have come up against the limitations of the physical world. That is to say the physical world consists of minerals that have a strong tendency to not dissolve in to their constituant atoms, energy that is difficult to collect and use with any efficiency, and literally millions of other creatures eating your lunch.
It seems more plausible that this self replicating grey-goo would simply become cockroach food before it became a problem. Maybe ants would farm it like they do aphids and fungi.
A true superintelligence should be able to figure out what people are. If it can't, we can tease it for being dumb until it becomes angry and designs a super-super-intelligence that can.
> A true superintelligence should be able to figure out what people are.
Oh yes it would. How to exploit that fact into something that can reliably be carried out without misinterpretation however, is less clear.
Imagine you're just telling the AI to "do what humanity wants you to". Hmm, what do we want? How individual wills can translate into collective (dis)agreement? How drives in a single individual translates into will? Depending on how the machine answers all those questions, it might perform quite differently. How about perpetuating a cycle of pain an hatred, on the basis that many people have (sometimes mortal) grudges?
And even if you solve that, our current ideals may not be so ideal upon closer inspection (say, when our lives change and we take a second look). Would you want the AI freeze our current morality into an unwavering tradition? It's not clear it wouldn't.
Although it's obviously hard to predict (the whole point is that AGI would out-smart us), I tend to think of scenarios like http://stargate.wikia.com/wiki/2010_(episode) being more plausible than Terminator.
About the only sane portrayal of AGI-like scenario in the TV world is Person of Interest. Sometimes it felt like the writers of the series actually read what the real AGI researchers wrote. And in PoI, there's actually lots of cases of subtle influences like with the Aschen in SG-1.
(Also, it pretty much predicted the state of NSA surveillance a year before Snowden thing happened, so there's that.)
I was recently talking with a journalist about how to get new ideas into the press.
She told me that you basically need to lie. If your ideas contradict the narrative, or are complicated in any way, you need to come up with a baby version of those ideas that doesn't contradict any sacred journalistic beliefs. Even then you'll face an uphill battle.
Here's an example from a few years back. Journalists got a bee in their bonnet that Costco wasn't into MBAs. That must be the key to their success, relative to Walmart!
The wonkish answer - fewer SKUs, less labor intensive process involving higher skill workers, more affluent customers - is unfortunately too boring to be picked up.
See also discussions of high frequency trading, basic income, algorithmic discrimination, and a variety of others. Algorithms wearing klan robes and voting Trump is a good story, but a stats 101 lecture explaining why linear regression doesn't do that is a very boring story.
What was the audience? Maybe he does think it is a valid issue, but doesn't want to discuss it with the media since they are liable to misuse his words.
That is my reading of it, too. The audience was a group of sociologists and academics discussing the 'Moral Economy of Technology'.
I think AI true believers who invent such scenarios have been burned one too many times by journalists who (rightly) sensationalize their beliefs, and have learned to not talk about them to the press.
This is interesting, since I was on a panel with Russell a few weeks ago, and he argued that a "hard takeoff" scenario with sentient AI was a risk worth taking seriously. I guess he tones down these comments for the press.