
The Magical Rationalism of Elon Musk and the Prophets of AI - scandox
http://nymag.com/selectall/2017/02/the-magical-rationalism-of-elon-musk-and-the-prophets-of-ai.html
======
philipkglass
What's Artificial General Intelligence and how is it different from ordinary
machines with programmable logic? It's machine intelligence that will _do what
I mean_ when I give it a directive, like an intelligent human being.

What's the danger of AGI? _It will do what I say instead of what I mean!_ Like
a not-so-intelligent human being, or basically every not-strong-AI computer
program ever written. It'll have a theory of mind so advanced that it can
sweet-talk its way out of any virtualized prison humans put it in, and
simultaneously so deficient that it might interpret "please cure cancer" as
"please exterminate multicellular life."

The story of AGI-as-existential-threat seems approximately as full of mystery,
trembling, fear, awe, paradox, and nonsense as old time religion. (The
Paperclip Maximizer is the familiar story of a loop without a terminating
condition. The one plausible way to build a superhuman intelligence that's
actually malicious is to "upload" a real human brain, with all corresponding
abilities and quirks, and run it faster in emulation. That's _conceptually_
plausible; I don't believe in souls. But "uploading" a human is so far from
_engineering plausibility_ at this point that you might as well worry about
the existential threat posed by time machines.)

