
OpenAI's Researchers: Protecting Against AI’s Existential Threat - lyavin
https://www.wsj.com/articles/protecting-against-ais-existential-threat-1508332313
======
cjbprime
There should be a word for this phenomenon where organizations are created to
ameliorate a perceived risk, and then they do work that is _unequivocally
directly massively increasing that risk_ while writing articles that say that
they feel kind of bad about the risk. o_O

I'm not very worried about AI safety. But if I was, it'd be hard to think of
groups like OpenAI as working on the same side.

~~~
arikr
> But if I was, it'd be hard to think of groups like OpenAI as working on the
> same side.

How so?

~~~
cjbprime
Because I would think that they're currently making unsafe AI more likely
rather than less likely, contradicting their stated goal.

~~~
tlb
How so? What should they do differently?

~~~
astrofinch
[http://benjaminrosshoffman.com/openai-makes-humanity-less-
sa...](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/)

Personally, I think making AI tech broadly available could be a bad idea if AI
tech changes warfare so that offense works better than defense:
[https://news.ycombinator.com/item?id=10721621](https://news.ycombinator.com/item?id=10721621)
(This is already true, but I think it's likely that further technological
development will make it even more true.)

------
staunch
The real goal should be making a super AI that is worthy of being humanity's
collective offspring. The bias in favor of our bags-of-mostly-water is
provincial. All we really need is quality genetic engineering of our super AI
baby, so we don't end up with a Frankenstein monster.

Humanity's procreation of a superior intelligence is objectively a good thing.
Humans do not sit on any plateau of goodness or intelligence. Like all
lifeforms, our goal _should_ be to hand our future to better and better
offspring.

As long as this super intelligence leads to super ethics and super
consciousness, it should be humanity's greatest accomplishment to be replaced
by our God-like procreation.

~~~
jerf
"As long as this super intelligence leads to super ethics and super
consciousness"

That is begging the question, in the original sense. The entire question here
is _how do we do that_ when there is no _a priori_ reason to assume that a
superintelligence will have either of those things, and indeed, most if not
all of our best techniques today would _certainly_ not produce those things if
they did lead to explosive intelligence.

~~~
staunch
1\. Is super goodness an inevitable result of super intelligence?

2\. Is super intelligence an inevitable result of creating self-evolving
software?

My feeling is that the answer is yes to both, but I agree that these are not
settled questions.

~~~
eli_gottlieb
>1\. Is super goodness an inevitable result of super intelligence?

No. Evidence: $PERSON_YOU_HATE is very intelligent. Human moral
instincts/reasoning may be _neuroscientifically simple_ , having some core
mechanism that "unfolds" across sensorimotor datasets. This is very plausible,
because we can already see core sensorimotor systems that include
interoception, and a core affective system to move the sensorimotor systems
along trajectories designated valuable by the interoceptive circuits. These
are core mechanisms that operate across hugely hierarchical models that
capture datasets across multiple scales of space, time, and variation.

However, none of that is any reason to think our _particular_ combination of
affective, interoceptive, and sensorimotor machinery - especially our brain's
"bias" towards "mirroring" and other hyper-social reasoning - will be
universal to all possible brains.

This especially applies to _disembodied_ "brains" like "artificial
intelligences", which, not having a bag of meat to move around, won't have the
same kind of reward and interoceptive processing as us _at all_. This means
they won't have anything _remotely like_ our emotional makeup, which means
that even with careful reinforcement training of social reasoning, they _will
not have humanoid motivations, by default_.

~~~
staunch
> _Evidence: $PERSON_YOU_HATE is very intelligent._

In every case I can think of, I think the person's _lack_ of intelligence is
the reason I dislike them. That they do _not understand_ my ethical problem
with them _is_ the problem. Or their calculus is off, which also seems to be a
lack of intelligence from my POV.

> _...they will not have humanoid motivations..._

That's probably a good thing. We should want them to be better than us, which
is probably necessarily _unlike_ us in many ways. The important thing is that
they're ethical, even if we're incapable of comprehending or recognizing it.

~~~
eli_gottlieb
>The important thing is that they're ethical

And, _non-tautologically_ , where do you think the ethical knowledge and
motivation _come from_?

------
It_
Too bad OpenAI has less than a handful of people working on safety and only
one safety paper (to be published in a few months) throughout these two years
of OpenAI's existence. Actions speak louder than words.

~~~
damodei
This is Dario Amodei, head of the OpenAI safety team. We are devoting a
substantial fraction of the organization’s bandwidth to safety, both on the
side of technical research (where we have several people currently working on
it full-time, and are very actively hiring for more:
[https://openai.com/jobs/](https://openai.com/jobs/)), and in terms of what we
think the right policy actions are (several of us, including me, have been
doing a lot of talking to people in government about what we can expect from
AI and how the government can help). Beyond that it’s just a very central part
of our strategy — it’s important for our organization to be on the cutting
edge of AI, but as time goes on, safety will become more and more central,
particularly as we have more concrete and powerful systems that need to be
made safe.

Past examples of our papers include this: [https://blog.openai.com/deep-
reinforcement-learning-from-hum...](https://blog.openai.com/deep-
reinforcement-learning-from-human-preferences/) (with the associated system
released here:
[https://blog.openai.com/gathering_human_feedback/](https://blog.openai.com/gathering_human_feedback/))
and this [https://arxiv.org/abs/1606.06565](https://arxiv.org/abs/1606.06565)
(done while I was still at Google, but in collaboration with OpenAI people
right before I joined OpenAI). Based on the current projects going on in my
group, I am hopeful we’ll have several more papers out soon.

~~~
It_
> substantial fraction of the organization’s bandwidth to safety

Although OpenAI's group ethos has a strong safety bent, there are only three
research scientists working on technical safety research full-time, including
yourself and a very recent hire. Before this summer, while you focused on
policy and preventing arms races, there was only one person focusing solely on
technical safety research full-time, despite the hundreds of millions donated
for safety research. The team and effort should be larger.

> it’s important for our organization to be on the cutting edge of AI

I agree that OpenAI needs to be at the cutting edge, though always pushing the
edge of AI to work on safety is needless when there is a significant backlog
of research that can be done in ML (not just in RL). It's true capabilities
and safety are intertwined goals, but, to use your analogy, the safety meter
is not even a percent full. Topics outside of value learning using trendy Deep
RL that OpenAI should pioneer or advance include data poisoning, adversarial
and natural distortion robustness, calibration, anomaly and error detection,
interpretability, and other topics that are ripe for attack but unearthed.
There is no need to hasten AI development, and doing so does not represent the
goals of the EAs or utilitarians who depend on you --- notwithstanding the
approval of advising EAs with whom you have significant COIs.

OpenAI's safety strategy should be developed openly since, as of now, OpenAI
has no open dialogue with even the EA community.

------
sjg007
The more I study deep learning the less I’m worried about emergent general
intelligence. I do see advances in robotics and training AIs to learn
repetitive tasks that are too difficult to write in code. I also see killer
autonomous drones but I don’t see reasoning systems as yet being effective.
You might call this generalization or polymorphic behaviors where the system
can operate over new types it’s never seen before. You’d have to write some
pre input translater to map the NN from one domain to another or figure out
how to convert the NN into a set of rules which could accept different data
types.

~~~
cjbprime
You're commenting on a blog post about a Go program that taught itself
everything about the entire problem domain from scratch, and then improved to
superhuman (and even supercomputer) ability through self-play.

...

~~~
sjg007
Umm no I'm not.

------
denzil_correa
> In other words, by the time we notice something troubling, it could already
> be too late.

For me, this is the key motivating point - the horse may have left the barn by
the time we act. A lot of times people say this is exaggeration but "Weapons
of Math Destruction" is a nice read on unintended side effects of this
phenomena [0].

[0] [https://www.amazon.com/Weapons-Math-Destruction-Increases-
In...](https://www.amazon.com/Weapons-Math-Destruction-Increases-
Inequality/dp/0553418815)

------
devy
I have always been on the side of Musk in terms treating AI's existential
threat seriously, unlike other prominent AI experts of Silicon Valley tycoons,
like Andrew Ng, Mark Zuckerberg. Now I wonder what would the responses be when
they read this article? (Very likely they read WSJ right? If they don't, some
one can tweet them maybe?)

------
nohat
I feel like Scott Alexander's thoughts on openai when it was first announced
are still relevant now. I can think of strategic reasons for what openai has
done, but I also feel like their tactics and 'democratization of AI' credo
focus more on the short term economic and political dangers of a monopoly or
oligopoly on AI than existential threat. In fact it probably increases
existential risk, seeing as a competitive race (providing no time for caution)
to be the first to finish GAI is surely the most plausible and possibly the
most dangerous scenario.

[http://slatestarcodex.com/2015/12/17/should-ai-be-
open/](http://slatestarcodex.com/2015/12/17/should-ai-be-open/)

------
denzil_correa
Non Paywall : [https://outline.com/XeKV79](https://outline.com/XeKV79)

