
OpenAI technical goals - runesoerensen
https://openai.com/blog/openai-technical-goals/
======
Houshalter
I'm concerned that none of these goals involve AI safety. Instead all of their
goals are nearly the exact opposite, of accelerating AI technology as much as
possible.

Safety was one of the main goals they promoted when it was founded. 2 of the 4
authors listed have publicly spoken about their belief of AI as an existential
risk.

I'm not saying that a game playing AI is going to take over the world. But it
does demonstrate the risk - we still have no idea how to control such an AI.
We can train it to get high scores. But it won't want to do anything other
than get high scores. And it will do whatever it takes to get the highest
score possible, even if it means exploiting the game or hurting other players,
or disobeying it's masters.

Now imagine they succeed in making smarter AIs. And their research spawns new
research, which inspires new research, etc. Perhaps over several decades we
could have AIs that are a lot more formidable than being able to play Pac Man.
But we may still not have made any progress on the ability to control them.

~~~
astanway
Almost no one who actually works or has done serious research in ML is
genuinely concerned about "malevolent AI". We are so, so, so far away from
anything remotely close to that. Please stop trying to gin up fear and listen
to the experts, who uniformly agree that this is not something to be concerned
about.

~~~
saltvedt
"AI: A Modern Approach (...) authors, Stuart Russell and Peter Norvig, devote
significant space to AI dangers and Friendly AI in section 26.3, “The Ethics
and Risks of Developing Artificial Intelligence.”

[https://intelligence.org/2013/10/19/russell-and-norvig-on-
fr...](https://intelligence.org/2013/10/19/russell-and-norvig-on-friendly-ai/)

In addition, there is the AI Open Letter, which is signed by many "who
actually works or has done serious research in ML", including Demis Hassabis
and Yann LeCun. From the letter:

"We recommend expanded research aimed at ensuring that increasingly capable AI
systems are robust and beneficial: our AI systems must do what we want them to
do. "

[http://futureoflife.org/ai-open-letter/](http://futureoflife.org/ai-open-
letter/)

Are the experts concerned about a "Skynet scenario"? No. But there is
certainly genuine concern from the experts.

~~~
ckrailo
Slight nitpick at "devote significant space"... AI: A Modern Approach is a
1000+ page book, 3.5 pages is a footnote by comparison.

------
JoshTriplett
#4, solving multiple games with one agent, seems like a reasonably interesting
step towards more general intelligence. An agent that can play a new game
without any game-specific information other than "what can I do" and "what
should I value" starts to sound a lot more like an agent that can solve non-
game problems. Especially if that agent can play games that model real-world
problems.

------
ktta
I find it really interesting that Goal 4 is a game playing agent. Deepmind has
been focussing on this since the beginning[0] and actually has great progress
as far as Atari games go[1].

And DeepMind is able to use a single agent rather than a different specific
one for each game. I wonder if OpenAI wants to go in a different direction
although RL has considerable success. Whereas the other goals definitely need
a lot of work before they are "real-world" functional, especially Goal 3. But
of course that depends on their definition of 'useful'.

[0]:[https://www.youtube.com/watch?v=rbsqaJwpu6A](https://www.youtube.com/watch?v=rbsqaJwpu6A)

[1]:[http://arxiv.org/pdf/1312.5602v1.pdf](http://arxiv.org/pdf/1312.5602v1.pdf)

~~~
argonaut
You're mistaken, DeepMind uses a different agent for each game. What's the
same across games is the learning method. But the model is trained separately
for each game.

~~~
ktta
I feel like I'm a bit confused at what constitutes a model. You basically have
the same 'agent' to start off on the RL process. You can just group the
learning agents together and make it (easier to tell it) automatically detect
the game that has to be played.

I guess I wanted to say that there isn't a lot different to be done when an
agent is being trained for different games. And that makes it a general game
playing agent, doesn't it?

~~~
argonaut
If you group the agents together and call it a single "general" agent, the
research community (and pretty much everyone else) will call you out on your
BS. That's the difference.

~~~
ktta
Them calling 'out on my BS' doesn't make it any less of a general agent if you
train it sufficiently with enough games. If you consider yourself a 'general
agent' who can play games with reasonable scores, I will ask you this. If I
give you a new game you've never played, how will you score compared to your
favorite game which you played a lot in your childhood? With the favorite
game, you basically remember the gameplay and you use it when you play again.
So is this me 'calling you out on your BS' at your ability to play games
because you remembered your gameplay? The same way that I'm suggesting that a
general agent remember its gameplay and use it to make a general game playing
agent?

I really doubt we will see an agent which can be an expert at a game without
even doing some computations which can fall into a grey area which people
consider to be game specific computation and not general gameplay. The general
agent which you are thinking of, which can be the best at a game without any
thinking (about what its game playing process should be) is a fantasy. It will
definitely need to have a 'gameplan' which it can get by simulating the
outcomes without actually playing it.

~~~
argonaut
Humans can _learn how to figure our the score_ , can _learn the state /action
space itself_, and can _learn new games without external intervention_. All
those things are hard-coded into a system where you just string together a
bunch of agents trained separately on different games, and where another human
has to add a new agent for a new game. Humans learn the gameplan and any game-
specific features. We don't have programmers plugging it into us for each new
game.

------
feral
Question:

"Build a household robot" is high up the list. That doesn't seem inherently
'general'.

Certainly, people have been working on that for years; there are all sorts of
subproblems like vision, contextual reasoning etc.

It _could_ be treated as a general problem, requiring a lot of 'common sense'.

But a team which sets out to optimize that particular goal, could spend years
on relatively narrow tasks that get good performance returns on household
chores (e.g. developing version 10 of the floor cleaning algorithm), but don't
really make progress towards the problem of general intelligence.

For me, what was really interesting about the benchmarks that Deepmind chose
(the choice of a selection of Atari games) was that they were inherently
somewhat general.

Are you not worried that by putting a narrow domain fairly high up, you'll get
distracted by narrow tasks, rather than making progress towards what's really
interesting - generality? Won't it introduce tension to try and keep the
general focus in the presence of a narrow goal, where you can get good returns
by overfitting?

~~~
argonaut
This isn't a particularly meaningful issue.

The problem you describe of falling into the trap of brute-force optimizing a
narrow task also applies to the Atari games. In fact it applies even more-so:
it would be trivial for a lot of HN programmers to brute-force code an AI for
challenging Atari games that deep learning still struggles against (like
Montezuma's Revenge). But it would be nontrivial (difficult but still
possible) to brute-force code a program for many narrow household robot tasks.
You avoid this problem by... not hard-coding brute force solutions/heuristics!
The research community can smell BS very easily (HN, not so much).

A household robot is _substantially_ (probably an order of magnitude) more
general than Atari games, even for narrow tasks (obviously it is nowhere near
the vicinity of the generality of AGI). The perception problem is tremendously
more complex. The control/planning problem is similarly tremendously more
complex.

~~~
ktta
>But it would be nontrivial (difficult but still possible) to brute-force code
a program for many narrow household robot tasks.

How difficult are you talking about here? Similar to training game agents? I
really doubt it.

I feel like the training problem is VERY hard in case of real world random
object handling (factory like fixed, mechanical situation can be purely hard
coded and be much better than a human). In case of virtual games you can just
use a bunch of GPUs and accelerate the process. But it is a much more
difficult problem in reality. The grasping ability that we have with everyday
objects is a marvel once you try to make a computer to do it.

This might be of interest:
[http://spectrum.ieee.org/automaton/robotics/artificial-
intel...](http://spectrum.ieee.org/automaton/robotics/artificial-
intelligence/google-large-scale-robotic-grasping-project)

~~~
argonaut
Atari game agents are trivial _if you hard-code / do traditional brute-force
search AI_ because there is no noise in observation and no noise in control,
and the control is very simple (usually just up down left right, no torques or
anything physically complex)

This is the state of the art of "traditional AI" (not deep learning) robotics:
[https://www.youtube.com/watch?v=8P9geWwi9e0](https://www.youtube.com/watch?v=8P9geWwi9e0)

Most decent HN programmers could code an AI for an Atari game in a few weeks.

Again, I would encourage you to read the literature instead of speculating.

~~~
ktta
It seems like you've misunderstood what I'm trying to say. I'm saying your
statement

>But it would be nontrivial (difficult but still possible) to brute-force code
a program for many narrow household robot tasks.

is very wrong. It is not really possible comparing atari games to even the
same class of difficulty as household chores. You've just agreed with my point
and said that I'm speculating.

>Atari game agents are trivial if you hard-code / do traditional brute-force
search AI because there is no noise in observation and no noise in control,
and the control is very simple (usually just up down left right, no torques or
anything physically complex)

I never said that they are trivial. Although the point I'm making, again, is
that you can't say that we can brute force even narrow household chores. It
has a level of complexity - friction (which is a huge problem), elasticity and
even air flow can mess up the actions, and they lack the computing power to
account for everything. Whereas we have something called intuition (which I
may add everyone interested in AGI to properly read up on, starting with Brain
Games - S4:Intuition which is on netflix)

And it seems like you don't consider brute-forced solutions as proper
solutions. I agree with that, as will any one who has common sense and read a
couple of wikipedia articles. But RL is not exactly the brute forcing as we
think of it, although it might look like it. We all employ brute force
learning in our own lives, to whatever extent it might be, although our
feedback and thought processes are much more complex so we feel we are acting
out of pure intelligent deductions we make in our brain. We still need a
couple of 'brute force' attempts, although with the number of iterations we
need, you can't call them that.

I suggest you read some literature too, and please point out where I'm
speculating.

1\. DeepMind's reinforcement learning paper :
[http://www.readcube.com/articles/10.1038/nature14236?shared_...](http://www.readcube.com/articles/10.1038/nature14236?shared_access_token=Lo_2hFdW4MuqEcF3CVBZm9RgN0jAjWel9jnR3ZoTv0P5kedCCNjz3FJ2FhQCgXkApOr3ZSsJAldp-
tw3IWgTseRnLpAc9xQq-
vTA2Z5Ji9lg16_WvCy4SaOgpK5XXA6ecqo8d8J7l4EJsdjwai53GqKt-7JuioG0r3iV67MQIro74l6IxvmcVNKBgOwiMGi8U0izJStLpmQp6Vmi_8Lw_A%3D%3D)

~~~
argonaut
If you're agreeing with my original assertion that household robot tasks are
more general and more difficult than atari games, great.

~~~
ktta
Almost. I'm saying the difficulty (of household tasks) is so much more than
they are in a different class of problems, and cannot be equated using a
comparative adjective

------
jimfleming
This is great! The goals seem reasonably ambitious and mostly doable over a
few years.

I am surprised by #2: "Build a household robot". It's my understanding that
efficient actuation and power are largely unsolved problems outside of the
software realm. What's the plan for tackling stairs, variable height targets,
manipulator dexterity, power supply, etc. in a general purpose robot with off-
the-shelf parts? (Answering these questions may be part of that goal but maybe
someone knows more on the subject.)

~~~
nightski
When it comes to AI/ML most problems seem "doable over a few years". Yet in
practice this rarely is the case. These are some extremely lofty goals. The
team behind OpenAI is quite capable but to say they'll have any of these done
in a few years is quite a stretch. I'm guessing they may achieve _one_ of the
goals in a decade. But I'd love to be wrong.

~~~
jimfleming
It was an off-hand remark. I'm aware of the landscape, though perhaps slightly
more optimistic. The first goal is simple enough and largely underway with the
Gym. Significant progress has been made on #3 and #4 just in the last year but
I agree that "a few years" is a bit brief. I remain doubtful about #2.

------
lowglow
Are the objectives of "OpenAI" conflicting with the interests of the startups
applying to YC? OpenAI is building their own products/platforms and insights
acquired from AI/Robotics startups applying and sharing information about what
they're working on might be used as a competitive advantage.

Is there an information firewall between what startups are sharing in the hope
of investment and what is shared to advance OpenAI? If there is a firewall,
how is it enforced?

------
fitzwatermellow
Goal 5: Inventing the Next Paradigm

Any impetus into actually dreaming up what may come next after GPUs and Async
DRL? Non-neural models, quantum computing based AI, optogenetic hacking ;)

Otherwise, excellent list!

------
mrfusion
I wish they had mentioned asimovs three laws.

------
nxzero
Is OpenAI willing to support true AI having basic rights to given to humans?

If so, why is this not one of the fundamental technical goals?

If not, why?

~~~
jimfleming
You should have a look a PETRL[0]: People for the Ethical Treatment of
Reinforcement Learners :)

[0] [http://petrl.org/](http://petrl.org/)

~~~
castis
That was way more of an interesting read that I thought it was going to be.
Thanks for posting that.

