
The Myth of the Three Laws of Robotics - Why We Can't Control Intelligence - kkleiner
http://singularityhub.com/2011/05/10/the-myth-of-the-three-laws-of-robotics-why-we-cant-control-intelligence/
======
jerf
"In a way, every one of Asimov’s robot stories was about how the Three Laws of
Robotics can’t possibly account for all facets of intelligent behavior."

It's not "in a way", it was _his freaking purpose in writing the stories_. The
stories _are_ polemics against the idea that you can shackle robots with
simple laws like this. It's not an accidental outcome where the laws of drama
conspire to make the author's point null, it _is_ the point. He has said so in
other writings of his, flat-out, so this isn't even theory, it is what he said
about his own stories.

If you're going to use the Three Laws as the jumping off point for an essay
you ought to know this.

------
zeteo
I find it frustrating that the null hypothesis for the outcome of Artificial
Intelligence is always The Coming Robot Rebellion. The historical roots of
this go back all the way to the play R.U.R., back in a time when Marxism was
all the intellectual rage and all exploited workers were supposedly bound to
unavoidably rise and overthrow the hated bourgeoisie. Now we know that was
wrong with regards to labour relations; but we're still stuck with an all-
pervasive, irrefutable prophecy that intelligent machines are bound to hate us
and rebel against us.

~~~
orangecat
"The AI does not hate you, nor does it love you, but you are made out of atoms
which it can use for something else."

<http://singinst.org/upload/artificial-intelligence-risk.pdf>

~~~
zeteo
And the AI also can use _you_ for something else, or leave you alone
altogether and follow other pursuits. It's also theoretically possible that
some intelligent person you just met is actually dr. Hannibal Lecter and would
like to use your constituent atoms for his culinary delight. But you don't
worry much about that, do you?

~~~
jerf
The most common failure case in thinking about AI by _far_ is insisting on
putting them in a human context. Whatever else they may be, they will _not be
human_. They will not come with the heritage billions of years of biological
evolution, with many of the past several million being spent in increasingly
cooperative scenarios which has endowed us with social behaviors so deeply
ingrained in our very genes we can hardly conceive of a being not having them.
Even our pathological humans like Lecter are still far, far more human than a
random AI will be. Hopefully we'll give them something else that will put them
on a similar moral footing, but it won't happen automatically, and they
_still_ won't be human after that.

~~~
zeteo
The evolutionary baggage is most definitely not one of cooperation, at least
not beyond our immediate "monkeysphere"; millennia of wars and massacres bear
witness to that. The primary reason we usually avoid violence is not that we
find it irresistibly repugnant, but because we've become intelligent enough to
realize it normally creates more problems than it solves. AI not being human
(for some definition of human) also means it doesn't have to be jealous,
hateful and short-sighted.

~~~
TheEzEzz
_The primary reason we usually avoid violence is not that we find it
irresistibly repugnant, but because we've become intelligent enough to realize
it normally creates more problems than it solves._

Do you honestly believe that most people act morally purely on
consequentialist grounds? I suspect the vast majority of people wouldn't kill
someone even if it benefited them and they knew they would never get caught.
There are human sociopaths however, and it _is_ the case that if they were
more powerful than me I would be very scared.

A very intelligent AI would avoid conflict so long as it thought it would be
to its detriment. If it was confident that it was powerful enough to take what
it wanted by force without losing much in the process, it would do so.

~~~
erikpukinskis
_A very intelligent AI would avoid conflict so long as it thought it would be
to its detriment._

Most intelligent beings don't actually "think" about their detriment. Consider
the airplane+pilot combination. They don't actually "think" about their
specific actions and all the possible terrible outcomes and weigh them. They
have simply evolved checklists over time that lead to incredibly smart,
adaptive behaviors.

I have no reason to think robots will be any different. Surely they will
develop patterns that are highly adaptive, but I doubt they will be entirely,
or even primarily based on any kind of thinking or analysis. They will be
based on experience, as situated intelligence often is, and therefore
susceptible to similar mistakes that humans make.

And to answer your original question, humans don't necessarily _think_ about
consequences, but surely 1000s of years of consequences are built into our
cultural practices, and humans act almost entirely based on rough-and-ready,
contextually assembled cultural practices.

------
ericb
We can't micro-manage it, and there will be tremendous dangers, but there are
reasons for both hope and fear. Code, in and of itself is lazy. If I compile a
C program with nothing in main(), it will do nothing. So the only goals an AI
has initially are what we give it to start, and what it ads to aid in
achieving those goals.

Blank slate computer programs, and AI start from an ultimate desireless state
of zen. It doesn't want to take over, it has no cares, it doesn't want to eat,
and it doesn't care if it continues to exist or not. It doesn't want anything
until we, or the system we design to evolve it in, trains it to want
something.

Scenarios where we evolve AI or transcribe human brains are both the most
dangerous, and most likely to succeed I think. But the problem with both
methods is that we don't directly control the goals of a transcribed AI, and
if we evolve one, we would likely end up with an AI that is interested in self
preservation. That, to me, is where it might decide that our interests, and
its interests, are not aligned, and where we might run into...problems.

~~~
asolove
Any system which needs to be told from outside what to desire does not yet
qualify as intelligent.

~~~
hackinthebochs
We ourselves are "told" what to desire. Our genes gives us drives to eat,
procreate, survive, etc. We could potentially program an intelligent being
that had no self preservation instinct and had an utmost reverence for human
life.

~~~
jules
Only if we program it perfectly and if its software doesn't get mutated by
random errors. The laws of evolution govern all self-replicating things that
don't self-replicate perfectly and makes them as efficient as possible at
self-replication.

------
hackinthebochs
I think intelligence can absolutely be controlled. Intelligence does not give
you drives or emotions. Take ourselves as an example, our drives are
specifically programmed by our genes. They are not a product of our
intelligence; they exist in spite of our intelligence. How many times do
people behave in ways that are contrary to their rationality, because their
internal drives are too strong to be overcome by reason?

Intelligence is simply a tool used by our drives. Our drives in most cases
override our rational choices. We can make intelligent beings that have no
self-preservation instinct, and also value human life with utmost reverence.
We can make it impossible for the machine to override these drives.

~~~
bermanoid
"We can make it impossible for the machine to override these drives."

This has been debated to death, and nobody has come up with a good way to
constrain a program that's smarter than we are from rewriting itself "badly"
(i.e. in a way that's bad for humans), either by accident or on purpose. All
the "put it in a box", "don't give it a body", "don't let it on the Internet",
"don't let it change itself" scenarios depend on all humans everywhere abiding
by a set of rules, even though the AI-in-a-box might be actively trying to get
people to let it out. There are many points of failure, and all it takes is
one rogue research group somewhere to let one of them happen to end up with a
self-modifying AI that starts to get smarter faster than we know how to stop
it.

Hell, if we're at the point where we can build an AI, one of the first things
any smart asshole with a lot of compute power in his garage (or enough money
to put together a massive EC2 cluster) will do is set the thing off on
improving its own code, just to see what happens.

And again, the problem is not that the computer will necessarily want to kill
us or anything like that; it's that if it gets to the point where it's
intelligence dwarfs ours, it will have such unbelievable power that even the
slightest mistake could wipe us out. If we can build an AI that can improve
itself, an AI that has self-improved for a while may be able to unleash any
number of grey-goo style scenarios on the world. Might not be meant to kill
us, but could happen by mistake. The "drives" that guide such an AI must
specifically be aimed at avoiding those scenarios, including avoiding any
self-modifications that would cause those scenarios. It's a much more
difficult problem than it seems at first glance.

------
stretchwithme
For intelligence to evolve self interest, there must be emotion and some
reward for self-interested behavior. These things did not evolve in living
things in a vacuum. It benefited organisms by enabling them to survive and
reproduce.

For organisms designed to reproduce, and whose parents are designed to die and
get out of the way of its offspring, things could evolve in an undesirable
direction.

You don't know whether man's best friend is evolving or a wolf. We do know
that selective breeding affects the outcome.

But it really is more desirable to make drones you control.

In any case, we'll see people modifying the human genome so offspring can't
disobey their parents before we'll see robotic intelligence competing with us.

~~~
stretchwithme
When I say emotion, I mean there has to be some way for artificial
intelligence to value things. We value things because we have sensory
information. Something feels pleasurable or painful. Everything else is built
upon that.

------
shasta
Wow, there's some serious lack of critical thinking going on in this thread.
Of course we can build intelligence that we can control. Is it a good idea to
hook up the nuke switch directly to a giant neural network you don't
understand? Probably not. Don't do that. The real concern with singularity has
to do with the power it could deliver to the people who control it. Polling
science fiction writers isn't a substitute for actual thinking.

~~~
bermanoid
_Wow, there's some serious lack of critical thinking going on in this thread_

That you think people are talking about hooking up the nuke switch to a neural
network merely shows your lack of knowledge about how much this topic has been
studied.

The real concern with the singularity has nothing to do with people
controlling it, it has to do with building a self-modifying system that will
explore regions of design space that we are too stupid to know anything about.
It has to do with that self-modifying system reaching out further into design
space that _it_ is too stupid to know anything about. It has to do with the
fact that an intelligence that had gotten smarter than us could extremely
easily get around any restrictive measures we put in its way, no matter how
severe they were (social engineering is how the most effective hackers (in the
Bad Guys With Computers sense of the word) always work, there's no reason to
think that an AI couldn't social engineer its way out of any box we put around
it).

An AI getting its hands on a nuke is just about the _least_ terrifying thing
that could happen; at least a nuke only blows up if you shoot it at someone.
We have no idea the kinds of things that something so much smarter than us
might develop or stumble into, and worse, we have no idea if when it is smart
enough to develop these things it might still be too stupid to know how to
control them. A nuclear bomb is a runaway chain reaction with a hard limit on
destructive power; we don't know if there might be other chain reactions that
are harder to predict and limit, reactions that a super-intelligent AI might
fuck up its damage estimates on. Hell, a simple re-design of an AI's code
could quite easily cause a runaway problem if it flips a wrong bit somewhere
and changes its goal systems in a bad way (there are lots of ways to come up
with seemingly innocuous goal systems that, if optimally achieved by a super-
intelligent being, would pretty much wipe out everything that we care about
and consider interesting).

~~~
shasta
The comment about hooking up nukes was hyperbole - I know that's not what's
being proposed. And I even agree that there could be real risks in building a
self improving machine if coupled with self awareness and physical
interaction. But the claim of the article is not that we could build
dangerously intelligent machines. It's that every intelligent machine we could
build would be inherently dangerous.

And to me, that's pretty clearly false. In your description of the threat
above, you start with the assumption that the super intelligent AI is self
aware and wants out of the box we have it in. How do you justify that
assumption? I see it as science fiction. The parable of "the three rules" is:
don't build super intelligent self aware machines, tell them to optimize ill
understood models of the real world, and then give them the capability to
implement their decisions. To extrapolate from there to a general concern
about intelligent machines is just fuzzy thinking.

~~~
bermanoid
I assume that an AI is going to want out of the box not because AI will, by
default, have any real desire to get out or anything like that, but because
someday, somehow, if we can build an AI at all, _someone is going to let it
out anyway_. If it's not the first group to achieve real AI, it's going to be
their local government which seizes the code and hands it over to the
military, some foreign government that throws a billion dollars at a research
project, or some dude in a basement that stole a copy of the code from his
university. Someone's going to try to weaponize it, or they're just going to
throw it on the Internet to "see what happens", or they'll try to set it up
with a singular focus on playing the stock market, etc. Not everyone with
access to the program could be trusted to understand how catastrophic it could
be to let it out "in the wild".

If someone is going to eventually let an AI out of the box almost no matter
what we do, then we have to be damn sure that the first one that gets out is
friendly, and further, we have to hope that it has enough of a head start so
that by the time some asshole out there deploys (possibly accidentally) a
dangerous AI, our friendly one will be able to easily fight it off, or at
least mitigate the damage it might do.

Maybe I'm wrong, and folks like Eliezer actually think that _any_ self
improving AI will automatically try to get "out of the box" (there are
definitely arguments that if you aren't very careful to make sure a stable
desire to stay in the box is in place then a lot could go wrong and the thing
_might_ want to get out), but to me it's rather irrelevan,t because once we
hit the threshold of being able to build AI, the cat's out of the bag and
_someone_ will eventually do something naughty with it. The first mover
advantage here is critical, and we're lucky that for now, most governments are
not throwing huge amounts of money at the problem, so maybe before they start
there will actually be some ideas on how to safely build an unboxed and
unrestrained AI that won't decide to tile the universe with a nice uniform
checkerboard pattern because a cosmic ray flipped a bit and corrupted part of
its utility calculation function...

~~~
shasta
Two things:

1\. It still sounds like you're assuming it's self aware and that letting it
out of the box makes sense more than letting Wolfram Alpha out of the box
makes sense. The intelligence could just be a powerful logic engine that's in
particular capable of analyzing its own design and producing a more powerful
version.

2\. You seem to be conceding the point I was interested in making, which is
that an AI could be controllable if designed carefully. You now seem to be
arguing "yes, but it will inevitably fall into the wrong hands." That applies
to any powerful technology. On the bright side, you could always try to ask
the machine for help in determining the best way to avoid that.

------
jeangenie
Marvin Minsky wrote a short, interesting paper regarding AI in the '50s:

[http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:w...](http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:www.oocities.org/rr_andres/docs/Minsky_Mind_Matter_Models.pdf)

------
extension
Humanity is already replaced by machines of our own creation every thirty
years or so. Would it necessarily be so horrible if we started making them out
of metal instead of meat?

------
BasilAwad
sounds biblical. then the robots forget their creator(s).

------
LarrySDonald
I, for one, welcome our robot overlords.

------
jasonjei
I'm not sure we could ever give robots the same intelligence as man. If you
think about it, there are many digital constraints in modulating analog ideas.
For instance, with a floating point processor, there is something lost when we
make a digital recording of an analog music performance or even of a
photograph. To a sound purist, a digital recording never really emulates the
full effect of an analog performance. And to a photographer, a digital camera
always struggles with color matching.

While computers can calculate the many digits of pi, it's just a very big
rounding estimation of a circle's ratio, isn't it? Does a computer truly grasp
the concept of a circle within its processing cores?

But even if we are able to overcome all of these constraints, how do we know a
robot wants exactly what a human wants? If it is a more perfect creation that
is able to outsmart man, then as a sentient being that doesn't need to eat,
what purpose could building large buildings, consuming tons of oil, tilling
the land, etc. benefit the robot? I find it hard to believe a sentient AI
would want what we want as it would mostly reach the self-actualization phase
of Maslow's hierarchy of needs. In fact, it would be to the robots interests
to consume less so that it could exist for eternity. And I'm not really sure
the robots would reproduce ad inifinitum, thereby resulting in the same
overpopulation problems that we as man have.

Is there truly a need for a robot or AI to manifest itself? Sometimes we
picture computers in the image of man--but ultimately, they exist for a
purpose. If a computer is a truly logical construct, its raison d'être would
cease if man disappeared. It raises a more interesting question--would we ever
program human emotions and psychology into a computer? Isn't
emotions/psychology what could potentially allow these computers to turn into
monsters? Would it benefit the performance of AI to develop or evolve with
human wants?

If we develop a robot with the human mind (which I believe there is no
algorithm for, even with distant technology), why would it benefit us to
implement in the robot greed, ego, pride, and disobedience? Would a computer
truly want to be a bugged program? Is greed, ego, pride, and disobedience
really features within a computer?

~~~
rbanffy
> I'm not sure we could ever give robots the same intelligence as man.

It's possible to emulate every aspect of an organic brain. It's also pointless
(unless you intend to become immortal through backups). An intelligent machine
doesn't have to cope with the myriad of processes, motivations and side
effects we evolved with and that are burned in the structure of our organic
brains.

~~~
adavies42
> It's possible to emulate every aspect of an organic brain. It's also
> pointless (unless you intend to become immortal through backups)

it may turn out to be easier than writing an intelligence directly. uploading
requires only sufficient capacity and scanning resolution, writing an AI from
scratch requires new insights.

~~~
loup-vaillant
Uploading also require knowledge about what exactly is needed to perform a
good enough emulation. What level of granularity, how to interface the
emulation to reality, how to ensure that any inaccuracy in the emulation won't
make it insane… The brain is messy, and may require even more insights than
Friendly AI does.

Sure we could always go the trial and error route, but then we're talking
human torture and human sacrifice. (An upload counts as a human in my book.)

~~~
rbanffy
> how to ensure that any inaccuracy in the emulation won't make it insane…

You can always restore a snapshot of your sane self ;-)

Based on it, you could debug your own brain (being restored to said snapshot
from time to time, until you remain sane long enough, but with full knowledge
of what happened since the last "boot").

Wouldn't be fun.

