
UC Berkeley Center for Human-Compatible AI - apsec112
http://humancompatible.ai/
======
idlewords
"Russell is quick to dismiss the imaginary threat from the sentient, evil
robots of science fiction."

This is interesting, since I was on a panel with Russell a few weeks ago, and
he argued that a "hard takeoff" scenario with sentient AI was a risk worth
taking seriously. I guess he tones down these comments for the press.

~~~
jessriedel
He may just think that "evil robots of science fiction" is not a reasonable
summary of the hard takeoff scenarios he worries about.

~~~
JoshTriplett
It really isn't.

If you have a full AGI, and it has the wrong value function, the scenario
isn't Terminator, it's "you've already lost and have zero hope before you know
what happened". Science fiction AI problems have absolutely no relation to
real AGI problems.

For one thing, a real AGI would have to hit a very narrow target (nearly as
hard as the one for success) to do anything _understandably_ evil, warlike, or
in any way recognizable to humans as an adversary; that would require
understanding human value systems and then acting against them. The far
broader target is simply that the AGI has a goal function, you're not part of
the goal function, and you're made of matter that isn't optimally organized to
meet the goal function.

And the reason why, despite that, AGI is not only worth contemplating but in
fact critically important and valuable, is that the success path for AGI
solves literally every single problem humans currently face, leaving only
challenges like "does the universe have a finite amount of entropy to make use
of?".

~~~
idlewords
My favorite part of this scary tale is the assumption that a general AI would
be highly motivated. I think Marvin the robot from Douglas Adams is a much
more realistic outcome of GAI than anything Yudkowsky or Vernor Vinge has
nightmares about. Or an autistic thinking machine that doesn't even want to
talk to us, and gets lost in contemplation.

Based on how the truly scary smart people I know behave, a GAI would spent
much its time panicking about meta-GAI, and how to make it safe for AIs like
itself.

~~~
sbierwagen

      My favorite part of this scary tale is the assumption 
      that a general AI would be highly motivated.
    

Largely by definition, we can't stop a superintelligent agent from doing what
it wants to do. We could build 9 strong agents in a row that just gazed at
their navels and proved graph theorems all day, but if the 10th one managed to
turn the "motivation" dial all the way up during hyperparameter tuning, then
it'll wipe us out in its quest to build more computers so it can prove more
theorems.

------
jedberg
For those that don't know, Russell co-authored the seminal book on AI with
Peter Norvig, who was also a Berkeley professor and is the head of research at
Google.

~~~
eschutte2
One of the few books I kept when I left school was a reference manual called
ANSI Common Lisp from Russell's CS 289. There was something special about the
author's writing - not only did he clearly "get it," he was able to
communicate "it" through written English. Years later when I heard about YC, I
thought the name Paul Graham sounded familiar, and sure enough, same guy.

------
miguelrochefort
I was expecting a rant about how natural language is an inefficient interface
for AI, and I got Friendly AI propaganda instead.

Does anyone buy into this at all?

It seems inherently anti-intelligent to limit AI to what we think is smart.
Are we going to make AI inherit our very limitations? That seems backward.

I don't see any objective value in artificially constraining progress as a
mean to preserve humanity. However, if I did seek to preserve humanity, I
would do so by coming up with a strong case for humanity's necessity, which I
would make available to AIs.

Thinking of AI in terms of anything other than Aliens is a mistake.

------
SticksAndBreaks
Human compatible? That sounds complicated, how about taking the set of In-
Human-Behaviour, and test for any new behaviour wether it is inherently in it?

------
CurtMonash
Joe Halpern is involved? Awesome! Great guy, although I haven't seen him since
grad school.

If nothing else, he may be the best songwriter I personally know. ;)

