
Superintelligence: fears, promises, and potentials - jonbaer
http://www.kurzweilai.net/superintelligence-fears-promises-and-potentials
======
Moshe_Silnorin
I don't think Bostrom or Yudkowsky believe this:

If one creates a human-level AGI with certain human-friendly goals, and allows
it to self-modify freely, the odds are high that it will eventually self-
modify into a condition where it no longer pursues the same goals it started
out with

I think their idea is a rational agent would have no incentive to change its
terminal goals, as changing your utility function has to be of negative
utility save for in some extreme edge cases that aren't likely to be relevant.
The hard part is giving it the correct terminal goal.

I also don't really think Bostrom is anthropomorphising. Basing your AI ideas
on Piaget's theory of cognitive development, that seems like anthropomorphism
- though likely entirely appropriate when the human mind is what you're
attempting to mimic.

Interesting article, brings back fond memories of Sl4,reading Ben, Yudkowsky,
Robin Hanson, Gwern, and two Satoshi candidates argue was quite fun. I was
convinced then that by 2015 nanotech or AI would have eaten the world. I'm a
little more uncertain now, but we still have a couple days.

~~~
joe_the_user
AI may be dangerous. AI can easily be dangerous without getting out of control
of all humans. Rather, it could be very dangerous in the hands of a malevolent
human - and we've not found a thing more inclined to harm humans and humanity
than certain humans.

That said, Yudkowsky's reasoning process about the situation seems
fundamentally flawed - both him and Bostrom imagine AI operating simply by
humans giving discreet, rational commands to AIs and having the AI twist the
wording of those commands, malevolently or just incidentally, to give rise to
terrible things - the myth of Genie writ large.

However, the failure of Gofia, of logical specification AI, shows us that to
be intelligent, AI would have to go beyond the level of just taking explicit
orders. Such an AI would "know what we mean" in the fashion that a person
would. IE, "rational" AI was discredited quite a while yet somehow the
"rationalists" have wormed their way into the position of interpreters of
future AI. Says more about human psychology than future prospects.

This doesn't mean other disasters aren't lurking but I'd say it means that
their particular disaster argument is untenable.

~~~
Houshalter
>both him and Bostrom imagine AI operating simply by humans giving discreet,
rational commands to AIs and having the AI twist the wording of those
commands, malevolently or just incidentally, to give rise to terrible things -
the myth of Genie writ large.

These are just hypothetical scenarios they write about, to help explain
problems with AI. I don't think they actually believe it will happen exactly
like that.

>However, the failure of Gofia, of logical specification AI, shows us that to
be intelligent, AI would have to go beyond the level of just taking explicit
orders. Such an AI would "know what we mean" in the fashion that a person
would.

Any truly intelligent AI would probably be smart enough to understand what we
mean with our words. The issue is that there is no reason it would _care_. In
the same way that if an alien showed up and started giving you orders, you
wouldn't really care, even if you deciphered his language and figured out what
he meant.

This is known as the Control Problem. The problem of controlling an AI's
motivations so that they actually want to do what we tell them to.

~~~
joe_the_user
_Any truly intelligent AI would probably be smart enough to understand what we
mean with our words. The issue is that there is no reason it would care._

Computer programs don't care about anything inherently. Assuming people
created programs with the ability to "know what we meant", the sensible thing
would be to program them to care primarily about "doing what we mean".

It seems like more or less an an anthropomorphization-driven illusion to
believe that attaining "human intelligence" cause a machine to have all the
unpredictable self-interesting-seek behaviors of humans.

 _In the same way that if an alien showed up and started giving you orders,
you wouldn 't really care, even if you deciphered his language and figured out
what he meant._

Except I'm not a tool (a thing crafted for a purpose) but a product of an
evolution with a combination of often contradictory impulses.

~~~
Houshalter
>the sensible thing would be to program them to care primarily about "doing
what we mean".

The problem is no one has any idea how to do this. We can make an AI with
desires. For example, we can give it a reward every time you push a button.
And then program it to predict what actions will lead to the most reward.
That's quite simple, because you can easily keep track of how many times the
button has been pressed in the program.

But given the opportunity, such an AI would just kill it's human creators and
steal the button, and hold it down. It doesn't care about anything but the
button. What you wanted doesn't matter at all.

How do you make it want to do what you _mean_. How do you directly measure
"obedience" in the same way we can measure a simple button press?

>Except I'm not a tool (a thing crafted for a purpose) but a product of an
evolution with a combination of often contradictory impulses.

To some extent, you are a tool crafted for a specific purpose. That is,
evolution crafted you to spread your genes as much as possible. And yet that's
probably not what you want. People have children, sure, but they also use
birth control, and do things other than trying to make as many copies of
themselves as possible.

~~~
joe_the_user
"We can make an AI with desires. For example, we can give it a reward every
time you push a button."

I wouldn't call that a "desire" in the fashion we've discussed it. I guess it
comes down to fundamental disagreement over how GAIs could be created. I think
it's obvious that the creation of a GAI would require a very careful
engineering of "understanding" in a broad sense - "knowing what people mean"
"caring about X" etc. in order to occur at all.

If just complexity and rewards are enough to create a thing that intentionally
increases resources and makes longer term plans, then it seems like we are
indeed in trouble. But I think that's implausible.

~~~
Houshalter
Call it a desire/goal/value/utility function/whatever. It's a thing which the
AI wants and creates plans to obtain.

This isn't that different than how animals and humans work, as far as we know.
We get pleasure from different things, and our brains seek actions which lead
to those things.

Creating an AI is only a matter of coming up with really good prediction
algorithms. Algorithms that predict the future reward the AI will get from an
action. While this is of course a difficult problem, it's not impossible, and
we are making a ton of progress on it.

Other models of AI all have similar problems though. At some level you need to
program the AI's goals explicitly. Even if that goal is "try to understand
what we mean, and do that", you still have to figure out a way of writing that
down in code. It's an impossible task.

~~~
joe_the_user
OK, I think I understand where you coming from.

While I don't think a GAI could arise from your scenario - just loops - I can
see now that you are making a plausible argument and given that the whole
field is extremely uncertain, it is reasonable to be worried by a plausible,
problematic scenario.

------
mark_l_watson
A good overview of all sides of the debate. It is a long article but it is
still a relatively quick way to get up to speed. Personally, I think the
danger from AGIs is minimal compared to the uncertainty and dangers in this
world - not now so much, but during the Cold War we faced the threat of
extinction. The world is relatively safe now and I don't worry about AGIs. By
the way, I think that the singularity and super intelligent AIs are in the
distant future, even with the accelerating pace of innovation.

------
haxel
What about the political implications of AGI? This concerns me most.

We need to carefully consider who exactly controls such a powerful tool and
what they use it for. Right now AGI is capital-intensive and risky: you need
large-scale computing, top PhDs, and plenty of time. This means that only the
most well-heeled organizations can justify earnest AGI development: big tech,
government agencies, etc. Those orgs already have plenty of political power,
and it seems to me that achieving AGI (or steps toward it) would amplify that
power. Perhaps to the point of being unchallengeable.

Do we want a political landscape where power is even more concentrated than
now?

~~~
argonaut
Right now, AGI does not _exist_.

~~~
haxel
Not that we know of.

If it does ever exist, how long will it take for us to learn of it? Powerful
tools are often kept secret.

And then, how long until the average person can leverage it themselves? The
most powerful tools tend not to trickle down, if it can be helped.

~~~
argonaut
Everything you've said is just speculation.

Super intelligent AI won't appear out of nowhere. There will be semi-
intelligent and normal-intelligent AI first.

------
HillaryBriss
The article lost me at "...it may well be best if advanced AI is developed in
secret by a small elite group"

How on earth could such a limitation be accomplished? It just doesn't seem
plausible. If AI offers some organization somewhere in the world a competitive
advantage -- how can development of that AI possibly be stopped?

~~~
chubot
Yeah it's not going to happen that way anyway. If you look at the history of
science, multiple discovery is the rule rather than the exception.

[https://en.wikipedia.org/wiki/Multiple_discovery](https://en.wikipedia.org/wiki/Multiple_discovery)

There are going to multiple groups... one of them could be small and elite I
suppose. But we have no idea in general how it will play out. People seem to
be scared of Google (OpenAI mentions them a lot), but the time frame is long
enough that that's far from certain, and perhaps unlikely.

I mean just like the Human Genome Project and Celera were within spitting
distance of each other. I think that's how it will play out with AI.

------
Udik
I think this is an excellent and thorough critique of Bostrom's
Superintelligence, which I've found to be a frustrating and sometimes
infuriating read. My impression of it is that the abstruseness of the language
conceals very few basic ideas, often repeated and poorly organized; vague
statements and a great amount of unexplored or underexplored possibilities. I
am actually very surprised by all the praise it has received. I'll probably
pick it up again and try to parse it a second time more carefully, but the
first impression was definitely that of an emperor with no clothes.

------
musha68k
The best "fail-safe" for such an intelligence IMHO: It should be able to act
out of _love_ and yes, naturally we'd have to earn it if we want to be treated
_humanely_ as well.

Love & understanding for _all_ fellow animals, _especially_ the weak and
vulnerable.

A whole new twist to the good old golden rule!

------
jbpetersen
I think we're far more likely to see superintelligence emerge as a result of
DAOs than as a result of AI.

