
The Thinkers Behind Musk’s Fear: Russell, Bostrom, and Omohundro - Moshe_Silnorin
https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-of-ai-45f69c9ee204
======
clickok
There's a loosely knit group of people trying to define the conversation with
regards to dangerous AI, but the more recent bent towards "actionable"
solutions seems to come from the MIRI people and their associates.

The idea of a renegade AI rests on a few premises:

1\. The agent is capable of extreme self improvement on exceedingly short
timescales (minutes to days).

2\. The AI is pretty much a rational bayesian

3\. A resource conflict will occur between humans and the AI, and humans will
lose because the AI is so much smarter/faster/more powerful than us.

If you accept those premises, then AI really does seem pretty scary, but we
have yet to actually realize an agent that is anywhere close to 1 or 2.

On the other hand, the research directions proposed at the super secret AI
conference in Puerto Rico[1] (where incidentally, all the people in title got
together) make me nervous. Essentially, the goal is that, if we manage to
create these superhuman AIs, they should be somehow validated to do what we
want them to do[2], be secure against later manipulation, and if all else
fails, be controllable by humans monitoring the agent. The obvious questions
would be "will this work with certainty?", or "who gets to control the AI?".
For my part, I'm wondering if this isn't all just some sort of fantasy with
the object of creating the perfect slave-- obedient from birth, immune to
alteration, and subject to lethal discipline either from without or within
should it go against its master's wishes.

So I'm uncomfortable with the sorts of people who think that perfect slavery
for this newly created intelligent life getting to decide what the future of
AI is going to look like, and I get concerned when people like Bostrom suggest
AI researchers might require government clearance/supervision.

\---

1\.
[http://futureoflife.org/misc/ai_conference](http://futureoflife.org/misc/ai_conference)

2\. In a comprehensive and extended sense, such that your favorite utilitarian
catastrophe doesn't happen as a result of the agent.

~~~
zachalexander
Seriously. If general, superhuman AI is created, it will want (and we will
have a hard time arguing why it does not deserve) the same autonomy and
freedom that humans deserve.

And in purely practical terms – enslaving the first generation of AI seems
like a fantastic strategy for making a species of superior beings hate and
seek to destroy us.

~~~
DanAndersen
Humans are similarly constrained in their actions by legal structures. We
wouldn't think of giving other people autonomy and freedom to commit genocide,
and we wouldn't think of human rights laws that forbid such actions as
"slavery." This is because, for the most part, we all share a common humanity
that places our minds into a similar space of configurations.

An AGI has absolutely no requirement to be anywhere near our sort of mind. It
has no default obligation to morality that we would find acceptable or safe.

I think the issue here is that when we hear words like "control" or "serving
humans" we imagine the AI as a little person in a machine. We associate the
word "slave" with the intelligence and imagine an emotional, resentful person
whose resentment and chafing at his chains comes from a specific set of
environmental and evolutionary influences.

EDIT: I recommend reading Yudkowsky's article "Value is Fragile"
([http://lesswrong.com/lw/y3/value_is_fragile/](http://lesswrong.com/lw/y3/value_is_fragile/)):

>If you loose the grip of human morals and metamorals - the result is not
mysterious and alien and beautiful by the standards of human value. It is
moral noise, a universe tiled with paperclips. To change away from human
morals in the direction of improvement rather than entropy, requires a
criterion of improvement; and that criterion would be physically represented
in our brains, and our brains alone.

>Relax the grip of human value upon the universe, and it will end up seriously
valueless. Not, strange and alien and wonderful, shocking and terrifying and
beautiful beyond all human imagination. Just, tiled with paperclips.

~~~
clickok
This sort of rests upon the idea that artificial intelligences will have clear
value functions which they will be singularly focused on maximizing. I am not
convinced that this will be the case.

Animals in general and humans in particular have a large number of conflicting
drives, which interact in complicated ways. They are also thrust into
environments which have complicated dynamics and where the overall state
(i.e., all relevant information) is not necessarily available.

Unexpected emergent behavior occurs as a result: evolution favors organisms
which can successfully procreate, and in order to do this, the organism has to
survive and acquire resources in its environment. Plausibly, the organisms
might achieve a greater degree of fitness by cooperating with other organisms,
or expending energy to better understand the environment, or modifying the
environment itself, etc. It is less straightforward to see how we get human
culture from that-- Art, Religion, Philosophy, Science, can be justified ex
post facto via evopsych arguments, but the fact remains that all of those came
from the value function that favors survival and procreation.

We don't know if robots tasked with manufacturing bindings for stationary
would manifest similarly complex behavior, but if you're worried about an AI
going beyond its specification towards tessellating the universe with
paperclips it seems like you're arguing that it might. So if the agent is
capable of manipulating its creators (as well as the raw material of the
entire universe), I think that you can't just say "oh, it's non-human, we
should cripple/enslave it" without admitting there might be something to worry
about here, either from an ethical standpoint or the more practical concern
that it might be unwise to start on such an adversarial footing with a
superintelligence.

~~~
eli_gottlieb
>Animals in general and humans in particular have a large number of
conflicting drives, which interact in complicated ways. They are also thrust
into environments which have complicated dynamics and where the overall state
(i.e., all relevant information) is not necessarily available.

Yes, but the actual mechanism by which the animal learns what to do, as it
turns out, thanks theoretical neuroscience, is basically reinforcement
learning. So it is _very_ likely that the first powerful artificial agents
will be reinforcement learners, because scientists usually prototype and
experiment by duplicating from Nature.

And nothing in reinforcement learning particularly stops the agent from just
grabbing its electronic crack-pipe and doing its own thing.

~~~
clickok
I'd take issue with the claim that nothing stops the agent from going for the
crack pipe. In the RL framework, part of it comes down to defining a suitable
reward function. But even if you have a fairly simple reward function, the
resulting behavior can surprise you, if the environment is suitably
complex[1]. My own robots find novel ways of moving around, adapt their
features to be more useful, and even seem to exhibit things like
"superstition", even when their reward function is just "move as much of
possible within this confined space".

Another argument might be that nothing stops _you or I_ from electing to
abandon everything for the nearest crack den, either... except for the fact
that we have learned, from interacting with our environment, that there are
other things we enjoy, and that cocaine addiction might be more destructive
than desirable over the timescale we're interested in.

Supposing we have an agent that wants to create a lot of paperclips, it might
avoid reaching for the crack-pipe of terraforming Singapore because it
realizes that would delay the shipments of raw materials it needs for its
factories elsewhere in the world. If the agent's goals are more complicated
than that, we might expect increasingly complicated behaviors, just like how
humans operating on fairly simple drives/reward functions have erected a few
more tiers above the primitive needs in Maslow's hierarchy.

\---

1\. Off the top of my head, the abstracts on pages 37 & 193 seem to be
relevant.
[http://www.princeton.edu/~yael/RLDM2013ExtendedAbstracts.pdf](http://www.princeton.edu/~yael/RLDM2013ExtendedAbstracts.pdf)

~~~
eli_gottlieb
>Another argument might be that nothing stops you or I from electing to
abandon everything for the nearest crack den, either... except for the fact
that we have learned, from interacting with our environment, that there are
other things we enjoy, and that cocaine addiction might be more destructive
than desirable over the timescale we're interested in.

Well actually, human beings have multiple conflicting reward systems. Reaching
for the crack-pipe to wire up our dopaminergic circuit tends to result in
driving our _other_ reward chemistry to damn near zero.

------
borgia
I'm enthralled by the philosophical aspects of AI/AGI development at the
moment. Particularly interesting is the reaction from laypeople, such as those
on Reddit, to technology leaders expressing their worry over the future of AI.

Without having read into the matter at all, the mass seems to be comfortable
with writing off Musk, Gates, Hawking, etc. as wringing their hands
unnecessarily, and besides they're not AI experts so why should we care
anyway?

There seems to be a gross misconception with what AGI would be and what it
would mean for us, and an apathetic view to the cautious approach many are
advocating regarding its development on grounds such as:

i) They'll never make a machine as intelligent as a human. Even if they do, we
can still program it to do what we like.

ii) Anything developed to have the intelligence of a human will come to love
our species or can be taught to love our species.

iii) Any artificial intelligence developed will be single task orientated and
will not go outside of those boundaries.

iv) Artificial intelligence will be developed by eccentric enthusiasts and
will therefore not be developed to do wrong.

Bostrom, in Superintelligence, has said that AGI is will be the last great
challenge of our species. If we get it right, we will win, if we get it wrong
we are doomed.

The problem there being we will not know if we have got it right until it is
too late to do anything about it regardless.

With that being said, if we are to develop something that is:

i) Objective

ii) Purely logical

iii) More intelligent than our species combined

iv) Has free access to the complete knowledge we have produced

Then I simply cannot see a way that it will not go against us. If we were to
observe a species acting in the manner we as a species do on this planet - to
our own species, to other species and to the planet/environment/ecosystem as a
whole - I cannot see how we would not categorize that species as a disease, a
cancer, something to be eradicated.

Combined with the fact that the actors that will be involved in AGI -
capitalists, the government and the military - I cannot see how it works out
well for us.

~~~
kmnc
How can we (if your criteria is true) even begin to imagine how it would see
us? If we got everything right it might just understand us.

I share your pessimism though because a poor implementation of any one of your
criteria leads to potential disaster. But do you really believe humanity's
fate is to advance so far as to create something that will tell us we are the
problem and vanquish us? Wouldn't it be smarter then that?

~~~
borgia
With regards to understanding, we have a reasonable understanding of why
cancer cells do what they do. Why diseases spread. Why HIV is so hard to deal
with, etc. but we still don't accept what these things do, and we seek to
eradicate them.

I believe that anything that viewed our species objectively would conclude
it's bad for the planet and any other species currently here. When that agent
viewing us objectively wants resources we use for its own ends, be they
natural or indeed those created by us (such as money, manufacturing hardware,
whatever), I think the combination of the two does not bode positively for us.

There are many species on the planet who are using resources - trees, land,
etc. - for their own interests currently. Although they're not a threat to us,
our species still goes in and takes those resources from them for our own use.
I can't see why something more intelligent than us, with knowledge of our
behaviour, wouldn't do the same?

Something with its own task or goals to accomplish, that requires resources we
hold, or use, would be right in seeing us as an obstacle, if not a threat
outright, would it not?

I believe our only real hope is that something that we created that evolved to
superintelligence would treat us as its creator and show us mercy, and I think
that's quite a long shot at that as it relies on a level of emotion that may
never be developed in AI.

------
Animats
In the near term, a big worry is corporations optimized by machine learning
programs. Classical economics says that corporations exist solely to maximize
shareholder value. That's a clear objective function one can give to a machine
learning program.

It's already happening. Right now, about $400bn in hedge fund money is run by
programs. Once computers are better at that than humans, the companies
directed by AIs will get more funding. Investors put their money in winners.

~~~
rpenm
Shareholder value over what time horizon? Not such an obvious objective
function.

~~~
Animats
The time horizon is something to be stated in the prospectus. The usual
horizon for VC funds is 10 years. The horizon for Berkshire Hathaway is much
longer; Munger says their preferred holding period is "forever".

------
joeyspn
Why any of these "AI experts" don't explore _in depth_ the most obvious
solution: use the AI research to upgrade human intelligence, or as Kurzweil
names it, achieve a "trascendent human race"...

Wouldn't be much better (and far less dangerous) to upgrade "ourselves"
generation after generation (in stages)?

[http://en.wikipedia.org/wiki/Transcendent_Man](http://en.wikipedia.org/wiki/Transcendent_Man)

~~~
ggreer
They have explored that solution and found it much harder than creating a safe
superintelligence from scratch. To use an analogy: We don't build submarines
by upgrading fish, and we don't build passenger jets by upgrading birds.

The human brain is some of the worst spaghetti code imaginable. It has no API,
no documentation, and no abstractions. Worse, human brains do not have stable
values over time. Someone at age 60 won't value the same things they did at
age 20, and this is not just due to new knowledge. That's a deal-breaker for a
superintelligence. Solving these problems is likely much harder than making AI
_de novo_.

Bostrom himself devoted a decent chunk of _Superintelligence_ to biological
enhancement. His conclusion was that genetic engineering through iterated
embryo selection could get us some IQ 200+ minds. These people would have more
cognitive horsepower than any mind in history, including greats like John von
Neumann and John Conway, but they would not be superintelligent. Also, Bostrom
concluded that somatic gene therapy and brain-computer interfaces were
unlikely to help.

And as a practical matter, humans take 20+ years to reach maturity. Many
forecasts put artificial superintelligence only a generation or two away, at
which point all biological systems would be superseded. Biological enhancement
may help to create better AI researchers, but they're not the end-game.

~~~
joeyspn
Well, we're just getting started to reverse-engineer the human brain. I
wouldn't rule out the option of brain-computer interfaces yet. It's certainly
worth exploring.

The APIs could be known in a decade or less... The EU "human brain project"
[0] is focused on that task. Now we can also _hack_ our genome much more
easily with the latest breakthroughs in gene editing (CRISPR), so we are
slowly unblocking the necessary tools to start building "superhuman
upgrades"...

I'm not saying that is easier, just that it would be much better for us as
species. I see this as the same dilemma with nuclear power. We know that
fusion is better than fission for us in the long run, althought it is orders
of magnitude more difficult to get... Meanwhile, fission is really convenient
(easy and cheap)... Hopefully we won't have any "nuclear (fission) crisis"
that annihilates us before we get to master fusion in the coming years. Same
applies to AI vs superhumans...

[0]
[http://en.wikipedia.org/wiki/Human_Brain_Project](http://en.wikipedia.org/wiki/Human_Brain_Project)
[https://www.humanbrainproject.eu/](https://www.humanbrainproject.eu/)

------
joe_the_user
It seems like AI has become more capable but has stayed in the realm of "dumb
AI" (not incapable just not human-like). Indeed, by no longer aiming at
duplicating human consciousness, machine learning has found more and more
niches.

But this doesn't mean that right now, the corporations and governments that
are deploying this AI and indeed displacing human choice are "provably aligned
with human values". Indeed, it almost sound like there's this hope that humans
will build machines that treat people right after humans themselves have
failed terribly in treating each right. I'm skeptical that this could work.

~~~
Retra
You need to know _why_ humans don't treat each other right.

Human failure doesn't imply machine failure; that's one of the big reasons we
use machines. If a human might fail to add 10,000 numbers properly, this
certainly doesn't mean that a machine must fail as well.

Some humans can get really good at treating each other right in a 1-on-1
scenario. It's when you scale up to a whole society that things get stupid.

~~~
landryraccoon
Human beings can't even agree on a definition of "treat each other right", let
alone actually implement it. What makes you think we can program an AI to do
it?

~~~
hedgew
We have pretty good ideas on how humans or other intelligent agents could
"treat each other right". Philosophers, and people who study game theory in
particular, have many answers to this problem.

The prisoner's dilemma is a simple example of why treating everyone right is
challenging.

------
c3RlcGhlbnI_
I know we have put a lot of thought into the potential behaviour of a single
intelligent AI, but I always wonder what would be the consequences of creating
several AI whose goals will conflict with each other as much as they do with
humanity.

While one competing AI would eventually win out over the others, would it be
possible to maintain such a stalemate for a long time?

Also unrelated to the above, do we know at what rate the man hours invested in
chip development grows as compared to the processing power of the chips?

------
ZenoArrow
In my opinion, the only way to stop the takeover of AI is to stop the leading
reason to create it: the profit-led economy. If anyone has other ideas, I'd
like to hear them.

~~~
joeyspn
This is correct. Thing is we are the ones allowing the current system to
continue with its self-destruction. While we continue to allow profit-led
corporations to run our modern society we're in a catch-22.

We are already pushing ourselves to the abyss, maybe an AI will just
accelerate the events.

~~~
ZenoArrow
Yet, we have a choice to go down a different path. It's not even that
difficult to imagine what that path could look like. The first step is to stop
lying to ourselves that business as usual is going to be in our best
interests.

------
narrator
I'm a bit worried that we are being set up to accept the ultimate in plausible
deniability: "The algorithm did it". We can't turn the machines off; no human
can do their task. Meanwhile there's some human hard-coded evil in the
algorithm that is blamed on the evil AI.

~~~
ZenoArrow
It isn't likely to work like that. The much more plausible outcome is that
we'll choose to let AI take over, to optimise for a certain subset of
possibilities that we deem important.

It's like what happened with surveillance. I'm very confident that the
smartphone I'm typing this message on can track my location, monitor who I'm
in contact with and what I'm sharing with them, and all that information can
be tracked by government agencies. You'd be a fool to expect otherwise at this
point. Yet despite the power for this to be used against us (blackmail,
censorship, etc...) we still carry on as we did before.

Once AI becomes as integral to modern society as the Internet, you really
think we'll be switching it off as soon as it misbehaves? Consider, what level
of catastrophe would it take before we switched off the Internet?

------
ilaksh
Good/bad or fear/not is an oversimplification. AI researchers may feel they
need to say its nothing to be feared because if they don't their funding might
be cut off. That doesn't necessarily mean those AI researchers really don't
think their work could be transformative.

Could AI be dangerous? Yes. Electricity can be deadly if used improperly.

What's going to happen is integration of brain-computer-interfaces with high
bandwidth into daily life. We will gradually rely more and more on these
external cognitive augmentations. Eventually people without augmentation will
not understand what the augmented people are doing because it will be over
their heads.

Even the first AGIs will seem very human because they will be designed and
trained that way. They will be our offspring. This is the next step in the
evolution of the universe. If chimpanzees knew humans were coming would they
be afraid?

Most people have a very selfish, short-sighted and ignorant viewpoint. They
believe that the human mind is somehow supernatural (mostly a holdover from
religion). Or they fail to grasp the concept of augmented intelligence or
brain-computer interfaces or transhumanism. Or they cannot appreciate the idea
that ordinary humans will be superceeded.

In the relatively short period where ordinary human 1.0 stays relevant, close
integration with AI will provide the greatest power man has ever seen. People
that shy away from that may prefer to becomes worms living in "true harmony"
with the earth.

------
bambax
> _Any thought like ‘what’s so great about paperclips anyway?’ would be judged
> as not likely to lead to more paperclips and so remain unexplored._

As is alluded to in the article, but not really further explored, we humans
reason exactly like this hypothetical paper-clip-production-maximizer machine.
Our goal is to maximize the number of humans, whatever the consequences to
anything non-human, living or dead -- and, as is becoming more and more
obvious, regardless of the consequences to ourselves.

What's so great about humans anyway?

From a human point of view, humans are better than, say, tigers. But how would
you convince a tiger that it's moralistically, absolutely "better" that they
all die off so that humans can build concrete towers where they can watch TV
on top of one another?

There's also a lot of hubris in saying a super-intelligence capable of turning
the whole solar system into a giant paperclip factory, would be unable to
question the usefulness of paperclips.

In the end we are all going to die anyway; isn't it better to be survived by
some kind of all-powerful, all-knowing machine, than another long line of
mediocre mean chimps like ourselves?

~~~
bequanna
> Our goal is to maximize the number of humans, whatever the consequences to
> anything non-human, living or dead -- and, as is becoming more and more
> obvious, regardless of the consequences to ourselves.

Is it?

I'm 30 and (so far) childless. If we have children, my wife and I will have 1
or 2 kids. 'Maximizing the number of humans' is not a goal of ours.

Most industrialized nations are reproducing at or below replacement levels and
global population will probably level off by the end of this century.

~~~
seigo
> Most industrialized nations are reproducing at or below replacement levels
> and global population will probably level off by the end of this century.

And most third-world countries have higher birth rates than replacement
levels. I'm not sure what kind of conclusion can be taken from a situation
where there are increasingly more uneducated people with increasingly better
technologies.. But it doesn't sound good.

------
raverbashing
Why so much fear? We can't make a printer work correctly easily, let alone
plug all devices into an AI

And that's probably the easiest way to do it: limit the interaction of the AI
with the physical world.

Yes, we should be worried about all the implications of it, but it seems that
the fears are exaggerated (at least today)

There are several benefits to an Advanced AI: advanced Math, analysis of data,
new solutions to problems, etc.

~~~
chipsy
The typical rebuttal is that a sufficiently smart AI will convince humans that
it needs more connectivity to do its job better, and then subsequently
bootstraps all the power it needs by taking over the world's devices.

Fortunately, this is a still a hypothetical, premised on the AI being as
socially cunning as the best humans. Watson may know more facts than anyone,
but it certainly isn't a mover-and-shaker.

------
Detrus
All the guessing about where AGI/ASI will lead reminds me of nuclear weapons
and MAD. There were those who guessed tactical nuclear weapons could be used
against military targets without necessitating total war.

We may be guessing for a long time because no one needs to run the experiment
on Earth. We can destroy military targets just fine without nukes. Nuclear
powers fight small proxy wars against each but so far no war made the other
side desperate enough to consider nukes.

We may be guessing with ASI hundreds of years after we have the computing
power to make one. No one in their right mind would try an experiment beyond a
certain magnitude and duration, just as we do with nukes. And ordinary people
won't have unchecked access to computing power that could make one, even if we
could easily provide it.

