
Thoughts on OpenAI, reinforcement learning, and killer robots - math_rachel
http://www.fast.ai/2017/07/28/killer-robots/
======
itsrishi
I can't speak for OpenAI as I know very little about it, but I 'm more than
half way thru part 1 of the fast.ai course and it's truly been a joy so far.
Thanks Rachel and Jeremy!

------
jph00
Rachel and I (fast.ai co-founders) will be keeping an eye on this thread, so
if you have any questions or thoughts, let us know!

~~~
nowarninglabel
Thanks Jeremy and Rachel for always sharing with the community!

------
chen19743
Does anyone else feel this is opertunistic? While I appreciate the online
course, why start firing shots at another non-profit? Seems the only realistic
answer is to drum up attention.

> One significant difference is that fast.ai has not been funded with $1
> billion from Elon Musk to create an elite team of researchers with PhDs from
> the most impressive schools who publish in the most elite journals. OpenAI,
> however, has.

~~~
jph00
It's because we care about our mission and want to support it and promote it.
We're entirely self funded from our own personal finances, so trying to
suggest we have some other purpose to our work is a little odd...

We want to bring attention to the important problems and people working to
solve them, and contrast that with things that are distracting from that.

------
AdeptusAquinas
I agree with this article: all the fear about AGI taking over the species
seems the hide the far more dangerous likelihood of efficient but non-general
AI ending up in the hands of intelligences with a proven history of oppressing
humans: i.e. other humans.

Besides which AGI, when it comes, is just as likely be a breakthrough in some
random's shed rather than from a billion dollar research team's efforts to
create something which can play computer games well. Not a lot Musk or anyone
else can do to guard against that, except perhaps help create a world that
doesn't need 'fixing' when such an AGI emerges.

~~~
unityByFreedom
> help create a world that doesn't need 'fixing' when such an AGI emerges.

I wish we would run with this rather than relying on tech and the market to
solve all our woes.

------
gone35
_I support research funding at all levels, and have nothing against mostly
theoretical research, but is it really the best use of resources to throw $1
billion at reinforcement learning without any similar investments into
addressing mass unemployment and wealth inequality (both of which are well-
documented to cause political instability), how existing gender and racial
biases are being encoded in our algorithms, and on how to best get this
technology into the hands of people working on high impact areas like medicine
and agriculture around the world?_

Not quite my field, but perhaps such currently intractable, high-impact
societal and medical problems _do_ require theoretical breakthroughs after
all... I guess I'm just concerned Rachel that --to put it in reinforcement
learning terms-- we need _both_ exploration _and_ exploitation.

~~~
jph00
Absolutely! We'd like to see investment in both. Currently we're just not
seeing the investment we think there ought to be in the areas Rachel
mentioned, or in application areas around food production, education,
medicine, and so forth.

~~~
icc97
Perhaps obvious question - but have you tried approaching the Gates
foundation? They are more concerned with sorting the issues of inequality.

Or, to be slightly obtuse, have you tried approaching Elon Musk for funding?
Perhaps he'll be a bit more flush with cash once the Model 3's start rolling
out the door and he'll have another billion around to donate. One of Musk's
OpenAI principles was trying to democratise AI and that sounds pretty much
exactly like what you're doing.

------
maxander
> ...but is it really the best use of resources to throw $1 billion at
> reinforcement learning without any similar investments into addressing...
> <important things>

The precise same sentiment could be made about his plans to go to Mars- people
are starving on Earth. But a Martian "backup civilization" _might_ someday
save humanity. Similarly, godlike AGI _might_ be mere decades away and
apocalyptically dangerous- predicting scientific advancements is generally
impossible, but some people working in the field don't think it's an
unreasonable possibility [1]. That humanity, via Musk, has put a billion
dollars into worrying about this somewhat plausible existential threat seems
reasonable, to me.

Musk is not a staid, Bill Gates-style philanthropist optimizing for dollar-
for-dollar benefits. He's picked a set of flashy, long-term goals and started
manic efforts to promote them. Perhaps it's a character flaw. But given that,
I think he's at least chosen _well._

[1] [http://aiimpacts.org/2016-expert-survey-on-progress-in-
ai/](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/)

~~~
icc97
Yes, I had very similar thoughts.

I think there's room for both. It's great what fast.ai are doing and they'll
benefit me more directly, but Elon Musk is pushing at the other end of the
scale.

I keep feeling like I'm a total Musk fanboy, but lots of his arguments that
I've seen make sense:

1\. The major one, is that doing things like aiming for Mars provides
inspiration for a better future. There's similar quotes from other people
involved with NASA.

2\. He talks repeatedly about improving the chances of better futures (clean
energy, hyperloop), and lowering the chances of bad ones (Bad AI, single-
planetary species on planet that has an extinction event)

He's put all his money into the things he sees as important, of which bad AI
seems to be high on his list.

------
philipkglass
_investments into addressing mass unemployment and wealth inequality (both of
which are well-documented to cause political instability)_

I don't really expect to see this solved in the USA before it's solved in
poorer countries. In the USA there's a tension between pleasing the millions
of surplus workers and pleasing the top billionaires that'll be minted when
robots can replace millions of workers. It's cheaper for the billionaires to
buy legislatures than to endure higher taxes or weakened IP rights.

By way of contrast, there are no local power brokers whose fortunes are built
on intellectual property in Bolivia or Bangladesh. It'll be 99% upside for
populist politicians to just ignore American IP concerns and freely "pirate"
machines-that-do-work and machines-that-make-those-machines. The long term
solution to machines-do-all-the-useful-work isn't to tax the profits and
redistribute them to unemployed people for spending anyway. That's just an
elaborate historical-theme-park imitation of a 20th century economy. "Machines
make all clothing, then we tax those machines and redistribute money to the
people so they can buy the clothing." The better solution is widespread
copying of the maker-machines and the dramatic price deflation that follows on
everything they can make.

~~~
visarga
The day we will invent a self-replicating factory that only uses cheap local
raw materials, the current economic system will end. We could say the whole of
the economy is a self replicating system, but we need to shrink that to a
small size and make it not dependent on rare or contested materials. Humans,
genes, and the ecosystem are self replicators as well. Self replication might
be a different kind of singularity that we reach even before AGI.

Basically, in order to make a self replicating factory we need advanced
3d-printing, robotics and a large library of schematics. Then, a "physical
compiler" could assemble the desired object by orchestrating the various tools
and the movement of parts inside the assembly line. If this automated factory
can create its own parts, then we have a self replicator. If you make it all
open source and ship seed factories around the world, soon everyone will have
their own stack to rely on.

~~~
Jach
I would rather have successful AGI before universal replicators that are
available to many merely human intelligences. Super especially hope for that
if these replicators operate at the nano scale, which I think you're implying
with the suggestion everything can be made from common raw material like dirt.
But I'm not too optimistic. Governments couldn't even stop proliferation of 3D
printed guns, how will they cope with other things in the future that have an
economic incentive too?

Don't forget that even if certain materials no longer become rare or
contested, location and raw energy will still be. Without a supreme AGI, or an
upheaval of our most basic knowledge of physics, expect wars to be fought over
rotating black hole real estate.

------
daly
Another subtle point (one of many) involves transfer of knowledge. My most
recent system "learned" by self-modification (similar in concept to growing
new synapse connections but done by self-modifying code). This kind of
modification will eventually arrive with Neural Nets as they move beyond
simple 'dropout' techniques to more context-specific additions of specialized
subnets and/or cooperating subnets (e.g. the room-recognizer adds a
'table'-specific subnet).

Assume two systems start out in a similar state. After interactions the
systems are no longer identical and they gradually diverge over time (think
twins). One is used in accounting and thinks of a 'table' as a kind of
spreadsheet. Another is used in woodworking and thinks of a 'table' as a
wooden object.

The implication is that one cannot just 'copy' what the first system knows to
'teach' the second system. There would have to be some sort of teaching
mechanism (aka college) to transfer the information.

------
grahac
Any timeline for when Part 2 is coming out? I checked your site and the URL
went to a 404. Loved Part 1 (it enabled me to do an ML project I had been
thinking about for over a year) and looking forward to part two! Please keep
it going!

~~~
jph00
Planning to release it on Sunday night :)

------
habitue
Personally I think it's absolutely important for some people to be working on
AGI and other to be working on practical applications. Trying to push the
boundaries teaches you a lot about what tools you still need. Researching
narrow applications hones our experience with the techniques we already
understand well and gives us new tools

------
blueyes
There is little to object to in this post, but there is also little to praise.
While it is an accurate description of the differences between OpenAI and
Fast.ai, it doesn't make a compelling argument that Elon and friends should be
spending their money elsewhere. Indeed, if the potential for neural nets to
help the world is so large, research into techniques like RL are by extension
even more promising, and should be funded and explored. Sure, neural nets are
being applied to all sorts of new problems of the sort we are familiar with.
This is a period for making incremental improvements in the paradigm. But that
should not obviate or preclude research that will lead to a paradigm shift. A
false dichotomy.

------
abhgh
> It is hard for me to empathize with Musk’s fixation on evil super-
> intelligent AGI killer robots in a very distant future.

This reminded me of a talk by Andrew Ng I watched recently [1]:

"Worrying about evil AI robots today is a little bit like worrying about
overpopulation on the planet Mars ... how do you not care ... and my answer is
we haven't landed on the planet yet so I don't know how to work productively
on that problem...of course, doing research on anti-evil AI is a positive
thing but I do see that there is a massive misallocation of resources."

Nice talk all around if you can find the time.

[1]
[https://youtu.be/21EiKfQYZXc?t=37m45s](https://youtu.be/21EiKfQYZXc?t=37m45s)

------
DrNuke
As for the intellectual challenge, I really think Francois Chollet says it
very well in his latest book: deep learning is a way to unpack, transpose,
visualise and re-aggregate data fundamentals. We only need to master a bit of
algebra and coding to make it happen on top of freeware wrappers like Keras
and Anaconda, to say two. All the rest pertains to domain knowledge, ethics,
politics and access to the hardware. Thanks Rachel and Jeremy for demystifying
this field for the masses.

~~~
chen19743
How did you get a copy?

~~~
DrNuke
From its website at Manning here [https://www.manning.com/books/deep-learning-
with-python](https://www.manning.com/books/deep-learning-with-python) ... 42%
discount code is deeplearning , as provided by Chollet himself... the ebook is
complete already, little work still needed to finalise before
commercialisation in October

------
mostafab
Reinforcement Learning is fundamental research, it needs sponsors like Musk to
survive. Agriculture and medicine are applied research, they can find their
own independent business models for monetization. That's what we are doing at
Startcrowd: [http://www.startcrowd.club](http://www.startcrowd.club)

------
mlinsey
"how existing gender and racial biases are being encoded in our algorithms"
sounds like a great first problem to work on for those worried about super-
intelligence, as it's basically a concrete instance of the value-alignment
problem.

IMO, as long as you accept that human brains run on physical processes (no
souls) and that computers continue to improve, super-intelligent AI is
inevitable; but it's reasonable to think it's more like 150 years away than
10. Given the magnitude of the consequences here, it's still worth spending
some resources to work on.

------
onuralp
I think Mark Nelson had succinctly described the problem with OpenAI (more
broadly, with the "AI safety" concerns):
[https://twitter.com/mjntendency/statuses/859189428748791810](https://twitter.com/mjntendency/statuses/859189428748791810)

------
esttime
What about using machine learning to estimate time to solve difficult task
with ai?

------
toisanji
it sounds like people who don't work with machine learning asked that
question. People who do work with machine learning should see the difference.
Besides what is mentioned in the blog post, OpenAI is a research effort for
moving start of the art, while fast.ai is for teaching students in a non math
heavy way.

~~~
jph00
We also do research and try to move the state of the art. Although we share
our results and methods largely through courses rather than academic papers.

Our plan is to teach more material each year, in less time, with less
prerequisites, by both curating the best practice techniques and adding our
own.

So far we've spent much more time on education than research since that's the
highest leverage activity right now (helping create more world class
practitioners helps move the field forward). And most of the research is more
curation and meta-analysis to figure out what really works in practice.

~~~
toisanji
Didn't realize you are doing research as well. I love what you have done with
the courses and look forward to seeing how the linear algebra computational
course does. anyway I could contact you to ask you more questions? My email is
in my profile

------
miguelrochefort
I have the intuition that, given a specific "dangerous" goal (i.e., paperclip
maximization), AGI will have too much autonomy to stick to it and AI won't
have enough autonomy to make it happen.

I believe that the fear of AI is unfounded.

~~~
PeterisP
Can you elaborate what you mean by "AGI will have too much autonomy to stick
to it" ?

Any system that is capable decide that e.g. paperclip maximization is a bad
goal must unavoidably have some scale of what constitutes better or worse
goals... and that de facto means that whatever is at the "good" end of that
scale will be the true goal of that system. But where does that scale come
from?

Especially given that this scale is _absolutely_ arbitrary, there are no
competing "innate drives" (like mammals have) that would make it difficult to
stick with any arbitrary goal. We're not talking about giving some
intelligence orders that might conflict with what it "really wants" \- we're
talking about configuring the ultimate desires of that system, this
configuration will define the world states that it will find more or less
"desirable" given complete autonomy.

------
test_hoobert
How would you compare the work that you are doing to the work that is being
done in China?

~~~
jph00
Can you be more specific? China is a big place and there's a lot happening
there! One non-profit in China for instance has kindly translated the entirety
of part 1 into Chinese and provided a discussion group for Chinese students.

------
Yuioup
Ok, so I consider myself an above-average programmer, capable of building a
standard database driven web applications using the latest du-jour techniques.

I suck at Math - I mean I _really_ suck at Math. I can visualize algorithms
and data structures and have no problem whipping up programs. I have written a
lot of code in my lifetime and have helmed a lot of successful projects as my
capacity as lead programmer or architect.

But I will never be a programmer that writes code that uses Math, like for
games, simulation, 3d graphics, operating systems, drivers, autopilots, etc...

So, can a Math idiot like me get into A.I. by going to your site?

~~~
j2kun
I'm working on a book for programmers who want to learn math. I can send you
the first few chapters if you're interested, but I'm also interested to hear
your thoughts about math in general.

~~~
gbersac
If you are doing this, this is what I'd love : a book with theory and ton of
exercises. My usual workflow is : I read theory, kind of understand something.
Then first try one type of exercise that applied that theory. I don't
understand a thing. Then I read the solution and I get it. I then need at
least two if not more exercises more to play with that theory before I REALLY
get it in my mind.

So please, add at least three exercises for each of the applications of each
concepts explained in your book.

~~~
pdimitar
Seconded 1000 times. We have tons of books written in a pompous manner with
self-patting on the back and full of theory only.

If you do indeed finish that book, consider me sold on it even if it costs
$150 -- if it contains a lot of practical examples and exercises.

------
daly
I've worked in a lot of AI-related projects and was around when the AI winter
arrived.

These various techniques that currently work by training, either supervised or
self-training, can have fatal flaws.

Take, for example, some high-tech camera technology. Use it on a drone to take
pictures of warships from thousands of angles. You take pictures of U.S.
warships, Russian warships, and Chinese warships. You achieve 100% accuracy of
identifying each ship using some Neural Net technology.

One day this net decides that an approaching warship is Chinese and sinks it.
But it turns out to be a U.S. warship. Clearly a mistake was made. Deep
investigation reveals that what the Neural Network "learned" happened to be
related to the area of the ocean, based on sunlight details, rather than the
shape of the warship or other features. Since the Chinese ships were
photographed in Chinese waters, and the U.S. warship that was sunk was IN
those waters at the moment, the Neural Net worked perfectly.

Recognition and action are only part of the intelligence problem. Analysis is
also needed.

~~~
jacquesm
That's called 'data leakage'.

[http://machinelearningmastery.com/data-leakage-machine-
learn...](http://machinelearningmastery.com/data-leakage-machine-learning/)

~~~
uptownfunk
Actually this is a clear case of overfitting leading to a high false positive
rate resulting in the misidentification of a US ship for chinese.

~~~
kgwgk
I don't think that's a clear case of overfitting. You could have used a subset
of the original data for training and the rest for validation and it would
have generalised pretty well.

It doesn't generalise when the US ship is in Chinese waters, but that's
because the system was never "learning" to recognize ships in the first place.

------
rrherr
Best paragraph:

"Cracking AGI is a very long-term goal. The most relevant field of research is
considered by many to be reinforcement learning: the study of teaching
computers how to beat Atari. Formally, reinforcement learning is the study of
problems that require sequences of actions that result in a reward/loss, and
not knowing how much each action contributes to the outcome. Hundreds of the
world’s brightest minds, with the most elite credentials, are working on this
Atari problem."

~~~
phreeza
Calling it an "Atari problem" sounds quite disparaging and misses the point.
It's like calling a convolutional network doing the ImageNet task a "Doggy-
detection" problem. That may be the original development problem, but the
final product still helps detect cancer in CT scan images... Same goes for
advances in reinforcement learning made on atari games.

~~~
jph00
Perhaps, but the jury is still very much out. The vast majority of RL
applications are game playing. Very few examples of valuable applications to
society or the economy.

There's also plenty of evidence already that RL isn't really the right way to
tackle the credit problem. E.g random search is only 10x slower.

~~~
cshenton
In what circumstances is it only 10x slower? Random search is totally useless
when your environment is stochastic. These algorithms aren't learning
sequences of actions, in fact most use a 30 'no op' random start to avoid just
that.

~~~
backpropaganda
By 'random', he means evolutionary search. It's not really random, and is just
a slower method for policy gradient. Here's the OpenAI blog post:
[https://blog.openai.com/evolution-
strategies/](https://blog.openai.com/evolution-strategies/)

------
bryananderson
Maybe I'm missing something here.

First of all, I know a lot of the popular perception of artificial "general
intelligence" is overly simplistic - I don't believe in the nerd rapture, and
I agree with a lot of what was written in the three articles tweeted by
Chollet that have been linked in this thread. And yet, I still don't see how
AI is not a plausible existential threat.

Unless you believe that some magical business is going on in the human mind
that isn't subject to the normal laws of physics, then I don't see how you can
believe that there's anything our brains can do that another machine can't.
Even if no cognitive skill can be increased to infinity, we have no good
reason to believe that our brains represent the maximum performance of all
possible cognitive modes. That's an appeal to the discredited idea that
evolution has a "ladder", upon which we stand at the apex as the finished
product. Natural selection doesn't optimize for intelligence, and it is not
"finished". So, if our brains are machines (albeit highly complex ones that we
only partially understand), and if they probably don't represent the maximum
potential performance of cognition, then how can we say with any confidence
that it is not possible to create another machine with higher cognitive
performance across all (or nearly all) modes of cognition? And if that is
possible, how can we say with confidence that such machines could pose no
existential threat to us? Sure, maybe they won't, or maybe we'll never figure
out how to build them. But how is it implausible? How is it something to laugh
out of the room?

Furthermore, AI need not surpass us in all modes of cognition in order to be
an existential threat. As AI gets better at accomplishing a wide variety of
tasks, it becomes an ever more powerful lever for those who own it. The near-
term threat from AI is socioeconomic: the replacement of vast numbers of jobs
with AI/robotics controlled by a small number of people who receive all the
profits of their "labor". It doesn't take much imagination to see how thie
could be, at the least, an existential threat to our current society if it is
not addressed with adequate forethought.

All in all, I just do not see how AI is not an existential threat worth
thinking about and sinking some money into researching how we can make it
safer. The tired old argument about the need to spend those resources on more
urgent matters doesn't hold water. There are seven billion of us. We can
specialize - indeed, it's arguably our greatest strength! We can - and must! -
devote resources and talent to a great many urgent issues, such as poverty,
conflict, disease, and illiteracy. But I think we would be very unwise not to
put a little of our wealth and time into researching how best to mitigate the
long-term threats that don't seem urgent yet and might not even come to pass.
If we don't, then chances are someday one of them will in fact come to pass
and we'll wish we had worked on it sooner.

~~~
jph00
I agree with everything you said. However, I don't agree it's what should be
hogging the money and interest right now. The impact of AI on employment (and
therefore societal stability) is a much more pressing problem. As is the
misuse of AI by people in many areas.

If society can get through these issues, then we may get to the point where
the existential threat of superhuman AI is of most immediate concern.

Either way, we're not arguing that this issue deserves no time and attention
at all. But currently it's the first (and often only) thing I'm asked about
when I give talks about the future implications of AI, and it's receiving huge
amounts of funding. Elon Musk, when he had the opportunity of addressing some
of America's most powerful people, elected to spend his precious time
discussing this issue, rather than anything else.

~~~
bryananderson
I definitely agree that it shouldn't hog all the money and interest - I just
don't think that it's hogging a disproportionate share right now. Musk and
Altman's OpenAI is a drop in the bucket in the universe of philanthropy. In
the AI field, the bulk of work at companies and research at univesities is not
going toward existential risk mitigation.

I think the "If society can get through these issues, then..." perspective is
fundamentally flawed. We will always, always have urgent problems.
Guanranteed. We need to work hard on them. But we also need to work a little
bit on the big things that are not of most immediate concern. I think a great
analogy for this is climate change. If people in the early Industrial
Revolution knew what we know now about greenhouse gases, they wouldn't have
been inclined to worry. The amount they were putting into the atmosphere just
was not significant, and even if it rose sharply, it would be many generations
before there _might_ be an issue. And yet, if people had put some focus on
long-term threats, we might be in less dire straits now.

I definitely sympathize with your experience as an AI educator being
constantly peppered with hypotheticals about superintelligence. I love your
fast.ai course and I am sure you'd rather talk about teaching AI than about
something that may or may not be an issue decades from now. But that doesn't
mean the issue doesn't deserve more funding.

Musk's topic choice may seem repetitive to us, as we are in the tech world and
read about/discuss this topic all the time. But a bunch of older politician
types probably have not even been exposed to these ideas of existential risk,
and as they hold a lot of power, it's not a bad thing to educate them on. For
what it's worth, Musk also spent a lot of time discussing nearer-term
implications like job displacement.

~~~
jph00
> _For what it 's worth, Musk also spent a lot of time discussing nearer-term
> implications like job displacement._

That's great to hear - I honestly had no idea. Which suggests that the media
doesn't have as much interest in covering that as it does the 'killer robots'
angle...

~~~
bryananderson
The transcript is definitely worth reading.

I think that's what has happened with this topic in general and Elon in
particular. In the talks I've seen him give, he seems less concerned with AI
being "out of control" and more with it being a very powerful lever under the
control of a small number of people. That's what he's given as the rationale
for OpenAI, and if you look at the research on their website, it's all real
research, not bloviating about killer robots. But of course the media just
wants to talk about killer robots.

------
backpropaganda
> Reinforcement learning: the study of teaching computers how to beat Atari.

This statement says more about the author and her inability to understand RL
than about RL itself. RL doesn't fit fast.ai's "AI is easy" narrative
therefore it's not worth doing.

~~~
jph00
There's nothing hard to understand about RL. I'm not sure where you get that
idea - if you find it hard, perhaps you just need to look at some different
way. Karpathy summarizes the differences with regular supervised learning in
his policy gradient post:

> _Policy gradients is exactly the same as supervised learning with two minor
> differences: 1) We don’t have the correct labels yi so as a “fake label” we
> substitute the action we happened to sample from the policy when it saw xi,
> and 2) We modulate the loss for each example multiplicatively based on the
> eventual outcome, since we want to increase the log probability for actions
> that worked and decrease it for those that didn’t._

(from
[http://karpathy.github.io/2016/05/31/rl/](http://karpathy.github.io/2016/05/31/rl/)
)

~~~
backpropaganda
I find RL hard and no I don't need Karpathy's or fast.ai's dumbed down version
of RL. I'm talking about cutting-edge RL like PSRL or RETRACE, not policy
gradient or DQN.

~~~
jph00
Well OK... I don't know why you think those are hard (let alone cutting edge -
PSRL isn't new), but it seems important for you to feel like you understand
things other people aren't smart enough to, so there's probably not much more
I can say.

~~~
backpropaganda
You can say RL is not hard when you manage to teach all aspects of the SOTA RL
algorithm
([https://arxiv.org/abs/1707.06887](https://arxiv.org/abs/1707.06887)) to your
class such that they are able to answer any question about it (not just
implement it). Good luck teaching metric spaces to code monkeys.

You guys are doing good work teaching tensorflow and algorithms/models
researchers are coming up with, but are slapping those same researchers by
disrespecting what they're working on now. Some humility would be wise.

~~~
pgodzin
Not sure why the exact details of state of the art research is relevant here.
Obviously that definition of RL is dumbing it down, as I'm sure Rachel knows,
but the point is simple - teaching a computer to do something specific a human
can without explicitly telling it whether something is good or bad, but rather
have it learn on its own.

The latest research in RL isn't getting us that much closer to AGI. We can't
plop a robot into the real world and tell it to use RL to learn everything.

~~~
backpropaganda
Before 2012, you also couldn't run a system to classify among 1000 classes.
Just because RL isn't there yet, doesn't mean it's not worth doing.

The reason we're discussing hardness of RL is fast.ai's narrative of "you
don't need math for AI" and "AI is easy". Sure, implementing and applying AI
is easy, and you just need to learn tensorflow, but doing even a modicum of
novel research in RL requires a tremendous background in all kinds of math. I
appreciate what fast.ai is doing to democratize as much of AI as possible, but
that doesn't need to be at odds with other people prioritizing RL research.

~~~
pgodzin
fast.ai's narrative is "AI is easy for what you probably want to use it for".
There are a ton of awesome applications that are enabled by the level of AI
taught in the course. However, Rachel's article is about how AGI is actually
really, really hard. So much so that we have no idea how to get there and
can't predict when it will happen. So instead of fearmongering about AI, we
should instead be encouraging everyone to do awesome new stuff with AI.

------
miguelrochefort
Regarding "killer robots":

If we can't make "paperclip maximization" the main goal of a some human, why
do we expect to be able to make it the main goal of some AGI?

~~~
PeterisP
We can't make anything the main goal of some human, because for us us (just
like any other animal) the goal system is genetically hardwired to a bunch of
complex feedback loops (hunger, sexual drive, pain, social status feedback,
etc etc etc), and we can't turn off any of them, much less all of them,
without fundamental alterations of something that's formed _very_ early in
evolution of multicellular life.

Any AGI system will also have _some_ set of goals, just like we do. For us the
goal is to (vaguely speaking) "feel good" and all the direct things that
affect that, including feelings caused by anticipation of future events, etc -
but this set of goals is pretty much arbitrary if these goals didn't have to
survive through millions of years of natural selection and orthogonal to
intelligence power.

I mean, if it doesn't have goals, then it won't _do_ anything, it's not an
agent. You could have very smart narrow systems that aren't "agentive" and
just e.g. provide answers to very complex questions, but whenever we talk
about _general_ artificial intelligence we are talking about a system that has
a feedback loop with the external world, i.e., it _does_ stuff, observes the
results, learns from that (i.e. is self-modifying in some sense) and decides
on further actions - which means having some goals.

------
nopinsight
Top people including DeepMind's CEO Demis Hassabis and Prof Stuart Russell, a
AAAI and AAAS fellow who is a co-author of the AI textbook used at most of top
universities, agree that AGI is definitely possible and going to happen. [1]

Hassabis also stated in another session that there are probably _at least half
a dozen mountains to climb before reaching AGI and he would be surprised if it
takes more than 20._ He also said that it has been easier to develop major
advances than he thought. [2]

Considering that there has been about one major leap per year in neural
networks architecture and systems for the past 3-4 years, and this includes
training neural networks to do one-shot learning for physical robot control
after practicing in a simulator, should we really be that complacent?

Even if the advances slow down to one per 2 years and it takes 12 of them to
get to AGI, that's only 24 years. If the rate continues to be one per year,
that's only 12 years.

[1] From the start of
[https://youtu.be/h0962biiZa4](https://youtu.be/h0962biiZa4)

[2] [https://youtu.be/V0aXMTpZTfc](https://youtu.be/V0aXMTpZTfc)

Note: Reinforcement Learning is far from the only component being researched
at major labs, including DeepMind and OpenAI.

~~~
tanilama
> Top people including DeepMind CEO Demis Hassabis and Prof Stuart Russell, a
> AAAI and AAAS fellow who is a co-author of the AI textbook most used at top
> universities, agree that AGI is definitely possible and going to happen.

Even though they are high-profile people, in DL, since people still don't know
a lot about it, their confidence means nothing.

When I was in college, my professors/textbook alike, claimed that in order to
conquer Go, we probably needs quantum computer or something really sci-fi,
maybe in 50 years, even 100 years. Guess what, no one, not even the most
optimistic person, would predict it will beat the best human player in 10
years. So, yeah, AGI might happen, maybe in 10 years, maybe in 100 years, but
until it happens, no one really knows when is that moment exactly.

~~~
nopinsight
The point is whether we should be complacent and dismiss concerns just because
we don't know when it will happen.

Once someone builds it, stopping it might be very difficult. Here's why:
[https://youtu.be/4l7Is6vOAOA](https://youtu.be/4l7Is6vOAOA) (less than 9
minutes and very clearly explained).

Dismissing even a 10% chance of possible catastrophic risks is not what we
practice in any other domains. Would you dismiss a concern over airplanes that
have a 10% or even just a 1% chance of crashing?

~~~
tanilama
Point is, nobody knows whether AGI really exists or not, based on our current
approach. As I have answered in another comment section, natural language
understanding haven't been cracked, at all. I mean at all.

So the reality is, it is like ancient human knows birds can fly with wings,
but people can only fly in their dreams. Now, we know people can think
intelligently with their brain, but no one knows how to make computer works
the same. It is far too early to talk about the risk, until we have the Wright
brothers of AI to enlighten us on such possibility.

~~~
nopinsight
Unlike airplanes, AGI possesses agency and the ability to cause widespread
harm and damage in today's computer-penetrated world. So if we wait until it
is invented, there is no guarantee that there won't be danger or even
catastrophes. Wouldn't you even agree that there is a 1% chance that AGI is
possible and that it could harm us once invented?

Also, the goalpost for identifying something as a challenging cognitive skill
worthy of the name AI is moved almost every time we make progress so people
keep denying that we are closer to AGI. AlphaGo is the latest example.

Some AI researchers and CS people believe that Natural Language Understanding
(NLU) is AI-complete, i.e. once we solve it we basically solve AI in the sense
of AGI. I do not personally believe that--There are certain human cognitive
skills that are not required for NLU. But I do think that solving NLU does
bring us closer to solving AGI.

Let's say someone makes progress on NLU. What would be the _minimum_ level
sufficient to convince you that AGI is possible? Why minimum? Because we don't
want AGI to be right at our doors before starting to prepare.

* Would getting _80% on Winograd Schema_ be sufficient?

Other suggestions are welcomed, including by others.

[https://en.wikipedia.org/wiki/Winograd_Schema_Challenge](https://en.wikipedia.org/wiki/Winograd_Schema_Challenge)

~~~
tanilama
For NLU, to convince me the potential, one important evidence I would be
particularly interested, is the demonstration of reasoning. Unlike the current
black-box model, I would love to see the model give explanation, along with
the answer, what it takes to lead such conclusion, be it a rule listed in the
manual or case that happened before.

This itself presents several big challenges: how do we represent the concept
of reasoning, in mathematical form? Knowledge required or not? What is the
representation of such knowledge?

A truly intelligent machine, should be able to take what human takes, a piece
of text, then build its knowledge base from there, then answer a question just
like human, then give explanation when asked for it.

------
supernintendo
> It is hard for me to empathize with Musk’s fixation on evil super-
> intelligent AGI killer robots in a very distant future.

I would argue this person has no business working with AI with this sort of
myopic thinking.

If you think "fake news" is a problem now, just wait until decentralized AI
networks are able to tweet, publish articles and affect public discourse in a
way that is indistinguishable from human influence. This AI could be trained
such that it pushes targeted propaganda and is able to destroy the lives of
dissidents using a variety of tactics (generation of fake audio and video
material that is indistinguishable from real life targets for the purpose of
slander [1][2] as well as other techniques like DDOS, hacking, etc).

Now imagine a world in which all devices are connected in an "Internet of
Things" and AI becomes sophisticated enough to exploit these networks and turn
them against humans. This has wide ranging implications from surveillance to
killing people by hijacking the computer systems in cars to even taking
control of military weapon systems that can be used to control and oppress
humans.

Now take all of that functionality and embed it within an actual robot that
can move around in the real world and is physically stronger than humans [3].

There is nothing that can be done about this because non-proliferation
treaties are impossible to enforce due to the nature of software. AI is going
to lead to the extinction of all human life on this planet and this is coming
from someone who generally disregards conspiracy theories as paranoid fear
mongering. We have every reason to be afraid.

[1] [https://arstechnica.com/information-
technology/2016/11/adobe...](https://arstechnica.com/information-
technology/2016/11/adobe-voco-photoshop-for-audio-speech-editing/)

[2] [https://www.theverge.com/2017/7/12/15957844/ai-fake-video-
au...](https://www.theverge.com/2017/7/12/15957844/ai-fake-video-audio-speech-
obama)

[3] [http://mashable.com/2014/01/12/robotic-
muscle/](http://mashable.com/2014/01/12/robotic-muscle/)

~~~
jacquesm
> I would argue this person has no business working with AI with this sort of
> myopic thinking.

What kind of thinking would you suggest would permit someone to 'work with
AI'? Should they be quaking in their boots before being allowed to work at the
altar?

You could argue (and with substantially more basis in fact) that Twitter and
Facebook have elevated each individuals utterings to broadcast status and that
even without decentralized AI networks all the effects that you are listing
are already present in the modern world.

It just takes a little bit more work but there are useful idiots aplenty.

Your SkyNet like future need not happen at all, what we can imagine has no
bearing on what is possible today and that seems to me to be a much more
relevant discussion.

> AI is going to lead to the extinction of all human life on this planet and
> this is coming from someone who generally disregards conspiracy theories as
> paranoid fear mongering.

Well, you don't seem to be able to resist this particular one. So your
'generally' may be less general than you think it is.

> We have every reason to be afraid.

No, we don't. I haven't seen a computer that I couldn't power off yet, SF
movies to the contrary.

~~~
afthonos
I would challenge you to power off Amazon's or Google's infrastructure.
Because a dangerous AI won't live in a desktop; it will live in a
geographically replicated system, across multiple power grids and legal
jurisdictions—all of which is possible _today_. And all these systems have
spent untold hours of very smart people's time to ensure they never turn off.

~~~
jacquesm
That doesn't mean they don't have off switches and it does not mean they can't
be turned off. I guarantee you the fire department knows how to switch any and
all of those off in such a way that the diesels and the no-break PSU's all
don't matter one little bit.

------
jorblumesea
> Cracking AGI is a very long-term goal.

Is this fair to say? I feel like advancements in the field happened far
quicker than anyone expected, and every few years we are reevaluating
timelines. Especially given the research happening in tandem that will almost
certainly speed up AGI, like graphene, quantum computing, advanced GPU
design...

If you asked someone 10 years ago about the possibilities of ML/Deep learning
they'd say it was far off too. I'm not going to say Kurzweil is correct but if
I know anything, it's that historically these things have happened faster than
expected. Look at 1997 -> 2017\. 20 years, but what isn't changed?

Appreciate any discussion as I am not an expert :)

~~~
jph00
We don't make specific time predictions because it's just not possible. But we
can make relative predictions. However long it takes to get to AGI, when we're
(say) half way there, the amount of societal upheaval will be enormous.

If we can't navigate that successfully, we'll never get to see AGI...

~~~
jacquesm
> when we're (say) half way there, the amount of societal upheaval will be
> enormous.

I had a conversation with a C level exec of a large company last week around
this theme. My suggestion that limited AI such as self driving cars has the
potential to create a vast number of extremely frustrated individuals making a
second round of 'Sabotage' and Luddites a definite possibility was waved away
as if those people don't matter and don't have any power.

I really wonder how far you'd have to be removed from the life of a truck or
cab driver not to be able to empathize with them and to realize that if you
take away some of the last low education jobs that allow you to keep your
hands clean that there will be some kind of backlash.

AGI will expand that feeling to _all of us_.

~~~
stale2002
If self driving cars/trucks become a thing, and cause millions of people to be
unemployed, there is ALSO a correspondingly massive economy increase.

IE, the world is now massively richer because of all this awesome new
technology.

Yes, there could be some short term disruption, but honestly I think things
will end up fine, just because of the massive amount of extra money and wealth
that the world will have that could be used to solve all those short term
problems that disruption causes.

~~~
jph00
That's not how things have generally worked out in the world. Billions live in
abject poverty even although the world has plenty enough to feed them and
provide shelter.

The issue is income distribution - and just making more money doesn't
magically fix the problem.

~~~
udkl
> just making more money doesn't magically fix the problem

Magically, no. With an effort yes. There are already experiments with basic
pay for example.

Jeremy Rifkin books such as [https://www.amazon.com/End-Work-Decline-Global-
Post-Market/d...](https://www.amazon.com/End-Work-Decline-Global-Post-
Market/dp/0874778247/) & 'Zero Marginal Cost' discuss the topic.

~~~
jph00
Yes exactly. We really hope to see this effort made. It's a fixable problem,
but often in history great inequality is only fixed by violent revolution,
which I don't want to see happen in my (or my daughter's) time!

