
Special projects - hendler
https://openai.com/blog/special-projects/
======
GuiA
_" If you do not work on an important problem, it's unlikely you'll do
important work. It's perfectly obvious. Great scientists have thought through,
in a careful way, a number of important problems in their field, and they keep
an eye on wondering how to attack them. Let me warn you, `important problem'
must be phrased carefully. The three outstanding problems in physics, in a
certain sense, were never worked on while I was at Bell Labs. By important I
mean guaranteed a Nobel Prize and any sum of money you want to mention. We
didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They
are not important problems because we do not have an attack. It's not the
consequence that makes a problem important, it is that you have a reasonable
attack. That is what makes a problem important."_

Richard Hamming, "You and Your Resarch"

[http://www.cs.virginia.edu/~robins/YouAndYourResearch.html](http://www.cs.virginia.edu/~robins/YouAndYourResearch.html)

~~~
lossolo
For most of this problems that they mention you need AGI, we do not know how
to attack this problem also.

Solving any coding challenge is NP-hard problem. You need to understand not
only what you need to do but also how language in which you need to do it
works. For example in game of Go you have huge amount of possible states but
you have only couple of moves to transition to this states. In programming
language every line of code has large space of states and it grows
exponentially with every new line of code. Sure you can brute force "hello
world" program but good luck with writing 1000 lines of code that work
together to solve complex problem. Pattern matching will not help you,
training based on github will not help you.

~~~
Darmani
Hi, I'm a researcher in program synthesis.

I think you meant to say "NP-hard" . And indeed, there are many program
synthesis problems which are NP-hard, 2EXP-complete, or undecidable. But there
are also many program synthesis algorithms which are polynomial time. If you
use Windows, there might even be one running on your computer.

We've been studying this problem for close to 50 years. I think you see the
basic problems, but we know a lot about what to do about them.

~~~
lossolo
Indeed, NP-hard (edited it now). Can you expand on "we know a lot about what
to do about them" ? Thanks.

~~~
Darmani
So first, NP-hard isn't much of a barrier. We often like to call it "NP-easy."
Or, going a step further:

"PSPACE-complete is great news. It's the new poly-time." \-- Moshe Vardi

SAT solvers are very fast. It's worst-case exponential, sure, but it's hard to
find that worst-case. I just heard at lunch yesterday that MaxSAT is now also
easily handling problems with millions of variables. I think I'd be scared if
I found out what the record was for normal SAT.

So, how do you actually do synthesis? I could talk for quite a while about
it.....but here's a pretty good intro-article:
[https://homes.cs.washington.edu/~bornholt/post/synthesis-
for...](https://homes.cs.washington.edu/~bornholt/post/synthesis-for-
architects.html)

~~~
tgflynn
_It 's worst-case exponential, sure, but it's hard to find that worst-case._

It's not that hard. Just generate a boolean circuit for the multiplication of
2 64 bit prime numbers and convert the circuit to a 3-SAT formula. I doubt any
current SAT solver can solve that problem. If you could do it for 1024 bit
primes a lot of cryptography would be toast.

EDIT: To be a bit more clear I mean that the circuit takes two n-bit numbers,
multiplies them then compares the result to some known product of 2 primes. So
by solving this circuit you factor the known integer.

EDIT2: Doesn't solving MaxSAT exactly imply that you can also solve SAT ? If
there's a SAT solver that can handle million variable instances "easily"
that's something I'd be really interested in hearing more about.

~~~
mortehu
The satisfiability problem doesn't require that you provide the solution, only
that you determine whether or not a solution exists. So you'd end up with a
primality check, which is known to be in P:

[https://en.wikipedia.org/wiki/AKS_primality_test](https://en.wikipedia.org/wiki/AKS_primality_test)

~~~
tgflynn
It's not a primality test. A primality test determines whether or not a single
given integer is prime.

The circuit I described checks that the product of two arbitrary input
integers is a specific known integer.

By solving the decision problem for each bit independently you can determine
all bits of the 2 input integers and hence factor the known target.

~~~
mortehu
Neat. I wasn't aware of the trick of peeling off a bit at a time.

------
superswordfish
> 2\. Build an agent to win online programming competitions. A program that
> can write other programs would be, for obvious reasons, very powerful.

The people on top desperately want us gone, and when it happens it'll happen
so quickly you won't know what hit you. Software engineers need to recognize
this common threat and organize (labor) sooner than later. Even more important
is for all engineers to plan for an imminent future where developers are not
paid like they are now, if at all. We have it good, but we will be automated
away like everybody else, just a little later.

~~~
BjoernKW
For simple data import / export tasks, simple CRUD, generic admin dashboards,
I agree that those tasks likely will be solvable by AI in the not too distant
future. As a matter of fact, I think it's somewhat bewildering we still need
human developers for that kind of task because the tasks themselves don't
really require a lot of intelligence. Problem is, the technologies and systems
we use to implement them today still require a lot of human interaction and
specialist knowledge.

Other than that, implementing an AI that is capable of understanding business
problems and solving them via code probably is nothing short of artificial
general intelligence. In that case we won't have to worry about not being
needed anymore anyway because by then we'll either very rapidly have a post-
scarcity economy or you know ... Skynet ...

~~~
superswordfish
We already have a post-scarcity economy. There is enough of everything to go
around for everybody. Kropotkin talked about this in "The Conquest of Bread"
_in the 19th century_. If history is a lesson, no degree of abundance will
result in collective plenty.

~~~
BjoernKW
> no degree of abundance will result in collective plenty

Will it? When you look at most First World countries in general nobody there
has to starve anymore.

In the Middle Ages only liege lords could get by without ever having to work.
Nowadays, even middle-class people don't necessarily need to work till
retirement age.

~~~
fapjacks
You and I have seen very different parts of the "First World", then.

~~~
nitrogen
Indeed, hunger is still a problem in the US. Our welfare system is so
fragmented and conditional that there are numerous cracks where people of the
wrong class/gender/neurotype/etc. can fall undetected and unhelped, and even
in cases where resources _are_ available, there is such a strong stigma
against using them that some people don't even realize they have the option.

~~~
ghufran_syed
I feel there are a lot of areas where poverty exists in the US, and as a
society, we should work to improve them. I disagree though, that 'hunger' is a
problem we should be trying to address. If we're claiming that a class of
people in society regularly don't get enough food, then I should see
malnourished people regularly in the hospital emergency department where I
work as a physician, and I _never_ do, except in the case of alcoholics, other
substance abusers, people with severe gastrointestinal disease, and some old
people with severe dementia living on their own (in which case, the problem is
their general inability to care for themselves, not inability to afford
sufficient calories). In fact, the most common nutritional problem seen among
the poor is _obesity_. Fyi, all the hospitals I have worked in regularly over
the last 8 years have been classified as 'medically underserved areas', which
usually corresponds very closely with most other measures of poverty, so I
don't think I am seeing an unrepresentative sample. Edit: I should have made
explicit, we _should_ of course be putting a lot more effort into feeding
those people who are malnourished around the world, of whom there are way too
many.

------
vonnik
I'm acutely curious how we can attack the problem of detecting _undisclosed_
AI breakthroughs, particularly those made by organizations uninterested in
ever revealing them. Commercially, yes, many organizations want to boast about
their AI, and we may find those AIs via games or news. But many organizations
involved in national security and financial markets will want to hide their
advantage. I would love to hear more about how Open AI thinks this problem can
be approached.

~~~
crypto5
By analyzing sequences of amazingly brilliant
developments/products/inventions/movements performed by single company or
conglomerate of linked companies, and which are few levels above what
competitors can deliver.

I am not from OpenAI though.

~~~
vonnik
By that measure, startups are powered by AI and large corps are not. By which
I mean, so many variables go into successful product development. Google
arguably has the strongest AI in the world, but it has been remarkably bad at
introducing new products recently, so those two phenomena seem decoupled to
me...

~~~
crypto5
And google definitely has few AI powered projects, others can't match. Google
is not a target of that goal, just because they always disclose their AI
achievements.

------
jesalg
This reminds me of an excellent two post series by Tim Urban about the
upcoming AI revolution: [http://waitbutwhy.com/2015/01/artificial-
intelligence-revolu...](http://waitbutwhy.com/2015/01/artificial-intelligence-
revolution-1.html)

------
Miner49er
For #3, wouldn't the AI defending against hacks need to be a hacking AI
itself? It would need to find vulnerabilities in the systems it protects in
order to plug them before other AI's find them.

~~~
tdaltonc
Yes and no. The analogy of 'attack' and 'defense' in hacking comes from the
military. So to extend that analogy, defensive missiles are still missiles,
but they are unquestionably different from offensive missiles.

~~~
msandford
The point is that in this analogy, no missiles exist yet. Nobody's gone to the
effort to figure that out. And then someone says "somebody might figure this
out an weaponize it! we gotta figure out how to neutralize it!" and then they
do the fundamental research to figure out how to make a missile at all. Now
you "just" have to swap out the guidance package and instead of a defensive
missile, you've got an offensive on.

Making an AI that finds vulnerabilities is the hard part. Doing an offensive
thing or a defensive thing is much easier if you've got an AI that's just
spewing out vulns. They don't need to be tightly coupled at all.

------
arcticfox
> 4\. A complex simulation with many long-lived agents.

Maybe someday we'll discover that this was exactly how the ancestor simulation
we're living in started.

~~~
brianchu
It's funny to see grown adults in tech taking the simulation idea seriously
just because Elon Musk said it. I was interested in science fiction in high
school and thought about these things, but I concluded they were worthless.

Why? It is _ontologically no different from_ 1) solipsism, a philosophical
idea that even most philosophers considered worthless centuries ago, or more
loosely 2) belief in a higher deity who created the world.

It's completely unfalsifiable. The answer to the question clarifies nothing
about our understanding of the world [1] and furthermore has absolutely no
impact whatsoever on how you ought to live your life.

[1] Even if you conclude we live in a simulation, we will _never know for sure
anything_ about the nature of the simulation or the nature of the world
outside the simulation.

~~~
dsacco
You can expand your reasoning to include all religion, while you're at it. And
anything anyone does that evidences a belief in something irrational or
unfalsifiable.

The thing is, you're not wrong. You're just kind of mean and arrogant to
reduce a multiplicity of nuanced worldviews that many people care about very
deeply to simple meaninglessness.

~~~
brianchu
Let's not resort to calling people names.

Your argument is ironically the reductive one. I have complete respect for
people who _choose_ to believe in an organized religion. But it remains the
case that whether or not you can prove a higher deity, that in of itself has
no bearing on how you should live your life. You can disprove the existence of
god and still choose to believe in religion. You can prove the existence and
still choose to not believe in any religious system. So proving the existence
is worthless.

~~~
dsacco
Fair enough; I don't know enough to say that _you_ are arrogant personally,
nor would that be productive. What I meant to express is that the comment
itself is an arrogant statement because it's self-centered. You directly
compared an idea people enjoy debating about and believing in to idol worship
(Musk) and something you dismissed as a teenager in high school.

This comment is actually a much more reasonable expression of your point than
the earlier one - you mentioned you respect the people who believe in religion
and revised your claim to state that _proof_ of the unfalsifiable is
meaningless (which is nearly tautological). That logic is sound (though you
could make an argument that attempting to prove something unfalsifiable when
you're emotionally invested in it and it's otherwise harmless could be
fulfilling).

Your earlier comment had a different tone and sentiment; namely, that _belief_
in something unfalsifiable has no impact on someone's life, and that you can't
understand why grown adults would bother with it. That was the specific
sentiment I found to be offensive.

For what it's worth, I'm agnostic.

~~~
brianchu
1) Discussion based on _rationality_ , of those topics, is worthless. Elon
Musk says he is almost certain we are living in a simulation, because of
_probabilities_.

2) You're the one calling it idol worship. I'm pointing out that most people
would not take this idea seriously if it weren't Elon Musk saying it - and I
still believe this is true.

3) The high school comment is pretty relevant. An intro philosophy class will
usually mention Descartes, who thought about this stuff 4 centuries ago.

4) I only said it's funny that grown adults take the simulation hypothesis
seriously. Your statement was the one that extrapolated that to religion and
found offense.

~~~
arcticfox
Ironically, _you 're_ the one engaged in seriously debating the merits (or
lack thereof) of the simulation hypothesis with people in this thread.

I was merely joking, it's simply pretty funny to imagine a full-grown world of
self-aware actors developing inside the OpenAI plan and then debating about
the simulation hypothesis.

------
nerdy
Programs writing other programs is an interesting concept, but isn't it naive
in most cases?

If program X can write program Y, program X is probably unnecessarily complex
with regards to solving problem Y. Program X could simply execute program Y's
operations at their abstract level, which would save it all of the outputted
language understanding and translation. The only tangible difference between
the two would be the ability to "save" the commands, at which point program X
is a scripting language and the input commands are the script.

Wouldn't this solution be better in most cases? If program X at runtime
requires significant configuration to produce program Y but those
configuration inputs are not saved. Then if program Y were to be changed,
program X must be rerun with all of the configuration input again. In the
scripting language scenario, the script is modified.

For a programming language to receive something simpler than the language it
is written in, the ideas must be abstracted to reduce the amount of code
required to perform the task. For example simple addition of multiple elements
can be reduced to a "sum" function. The "programmer" using this must still
have knowledge of the sum function and its use. Abstractions require the
system to have knowledge of that domain, so with each new domain a whole other
set of abstractions and complications are introduced.

So if a machine were to write the entire program, how does it differ from
machine learning? While machine learning might be often used for smaller
algorithms, this case would apply ML to the entire problem as sets of smaller
problems. If a human must still program in some abstracted sense, isn't it
just a scripting/programming language? How would a component of this suggested
project not fit into the "machine learning" or "scripting/programming
language" categories?

------
Cybiote
#2 Is incredibly ambitious but I expect it should be doable by a lone grad
student over the course of a summer. Something like this would range in
complexity from completely redefining programming languages (drastically
lowering the skill required for complex programming tasks and thus feeding
into #1) all the way up to general intelligence.

The tests referenced in
[[https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf](https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf)],
currently dominated by information retrieval techniques, seem more realistic
and still feel hopelessly far away.

#3 should also allow looking into the use of AI to break into systems, in
addition to detecting and defending against AI breaking into systems. A
prototype for #4 could create an environment where trilobyte fuzzers could co-
evolve into fearsome Artificial intelligences. The AIs that break into systems
need not be as smart as the systems being broken into. Just as viruses are
much less complicated than eukaryotic cells and yet are capable of wreaking
great havoc against mammals, might this be a possible mad deterrent against
out of control AI, developed by #1's opponents who use #2's breakthroughs? No
AI could plausibly be bug free.

See also: Schild's Ladder.

#4 It would also be cool if humans were allowed to visit and interact with
this virtual world.

See also: The Lifecycle of Software Objects

These projects have the same feel as: A PROPOSAL FOR THE DARTMOUTH SUMMER
RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE, whose #4 and #1[+] were the only
ones to see much progress. It's difficult to say whether we are 10 or another
50 years away from making meaningful progress on OpenAI's list, but _I 'm glad
they made it because it seems somewhere along the past 60 years, we forgot how
to dream_.

[+] #3 warrants a honorable mention.

~~~
skizm
#2 Might be technically doable: I feel like there has got to be a way to
"cheat" this. Maybe make something that Google searches similar problems and
then automatically signs up for contests under several names and submits
potential code snippet answers as solutions. Your program could run the
snippets in sandboxes and ensure the inputs / outputs match the example code
for the problem. You do this on a large enough scale you might eventually win
a few just by dumb luck + your skill at programming something to detect how
well random snippets of code fit a particular problem.

~~~
Cybiote
I think that sort of solution would work better as an educational tool or as
part of an augmented programming environment. But I don't think it fits within
the spirit of the task. It also would hit a ceiling really early on, if the
parallel solutions to the Allen AI Science challenge are to be looked at as
hints of how things might turn out.

------
tom_b
#2 - Build an agent to win online programming competitions.

Given that online programming competitions tend to fall into distinct classes
(dynamic programming, algorithmic challenges, graph problems, string
problems), this seems maybe more "solvable" with a non-AI implementation?

Imagine you have a framework that can spit out sub-pieces of a solution that
worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected
components in each graph | spit answer of graph with lowest number of SCC).

Then you need to grind out some type of expert system to replicate the
competitive programmer who currently chunks together those framework pieces.

Of course, this probably doesn't work at all for something like kaggle or
stockfighter.io, where parsing the instructions and observing a dynamic system
are key parts of the hacking. I am more thinking of SPOJ or similar . . .

~~~
nl
There's been significant progress in data science competitions (Kaggle etc).

Not so much parsing of the problem itself, but building generic frameworks
where - given data and a target variable - it will work out the type of
problem (eg classification vs regression) and develop a predictive model
without human intervention.

ICML has had workshops for the last two years on this problem[1][2]

[1]
[https://sites.google.com/site/automl2016/](https://sites.google.com/site/automl2016/)

[2]
[https://sites.google.com/site/automlwsicml15/](https://sites.google.com/site/automlwsicml15/)

------
swalsh
I love how much these things just FEEL like Elon Musk fantasizing about an
idea, and just really wanting someone to work on it.

------
freyir
The goals are an AI that can augment itself (project 2) and defend itself
(project 3). I've found Musk's fear of AI to be a little overblown, but this
is nuclear weapon type rationalizing: "AI could be end the world as we know
it, so we'd better race to create it first."

It's interesting that Projects 2 and 3 would make great businesses even if
they only got halfway to "existential threat" level AI. Then there's project
4, version 0.1 of Musk's "we're all living inside a simulation" claim. Musk as
creator of worlds.

------
deutronium
The security project reminded me of the DARPA Cyber Grand Challenge -
[https://www.cybergrandchallenge.com/](https://www.cybergrandchallenge.com/)

------
feral
> Detect if someone is using a covert breakthrough AI system in the world.

I bet the real purpose of this is not "It seems important to detect this", at
least against superhuman AGI.

As that's asking you to win a game against a more intelligent game player, who
is going to be able to out-think you and hide.

It seems more likely that:

The real purpose of this problem is to impose a 'secrecy tax' on entities who
would like to recoup investment by covertly using such a system, thus reducing
the incentive to build one in secret. I.e. if you know someone has invested in
research that's going to limit your ability to profit from covertly building
an AGI, this means you can only use it up to the limit of game theoretic
detectability, reducing the profit motive of acting covertly in the first
place.

Its a bit like once you break Enigma, you have to be careful how you use it.
Having alternative explanations for how you got your edge then becomes
valuable.

So, if you see a corporation building something that could provide an
alternative explanations for huge returns, other than AGI - ideally for
something that's very noisy (high variance) - and where it'll be hard to tell
whether their success was by chance or on purpose, that's probably (very weak)
evidence (in the bayesian sense) they are working on a secret AGI.

I suggest YCombinator is an ideal candidate... huge returns, very hard to tell
if its luck or skill.

~~~
mindcrime
_I suggest YCombinator is an ideal candidate... huge returns, very hard to
tell if its luck or skill._

Maybe pg is actually an artificial super-intelligence. I mean, has anybody
_actually_ seen him in real life???

------
endergen
5\. Focus on how you would use AI to augment personal human thinking.

------
sozforex
#1 - reminded me of "Endgame: Singularity"
([https://en.wikipedia.org/wiki/Endgame%3A_Singularity](https://en.wikipedia.org/wiki/Endgame%3A_Singularity))

------
auggierose
That stuff sounds so generic, not sure what is special about it :-)

------
akeck
I christen number 1 "The Ghost in the Shell Problem", in reference to the
project at the core of the first movie's plot.

------
pboutros
#1 - talk about distributed, automated, statistical analysis. What an exciting
project to get to work on.

~~~
gdb
> What an exciting project to get to work on.

Indeed :). Anyone who's excited, please apply! Feel free to ping me with any
questions: gdb@openai.com.

------
chewxy
Anyone who starts a project for #1 will get instantly targeted by Samaritan's
agents, right?

------
swah
Is this job application for humans only or AI can apply?

------
vonnik
Riffing on project no. 1, here's our post on "How to Control AI":
[http://deeplearning4j.org/skynet](http://deeplearning4j.org/skynet)

------
frank_jaeger
>A complex simulation with many long-lived agents.

No need to start a new one here, just download Dwarf Fortress.

~~~
outworlder
Except that DF Dwarves are, by definition, not long-lived.

------
thuruv
This will certainly backfire

------
jcoffland
Theses "ideas" sound like either they were written by a 14 year old kid or
someone very high. I was hoping for real science.

Yes I know my comment is probably against HN guidelines but there ought to be
an exception for making fun of uber rich Silicon Valley types. They parody
themselves. It reminds me a lot of the show Silicon Valley.

~~~
ericjang
care to elaborate why this isn't "real science"?

~~~
jcoffland
It's real science fiction. None of these problems are even close to being
practical except in the fantasy world created by pop-sci news outlets.

I was hoping for a serious discussion about the problems on the forefront of
machine learning.

~~~
superswordfish
Well that's why they're courting not just experts, but "strong experts" of
machine learning, whatever that means :P

~~~
jcoffland
It must be experts who have worked out in the OpenAI Gym. Apparently by
playing video games.

[https://gym.openai.com/](https://gym.openai.com/)

------
AndrewKemendo
edit: Boneheaded passion got ahead of me and I forgot that the Gym was
intended originally to be cross functional. TO be fair I don't think that
aspect of the gym has been well publicized. Original comment retained for
context:

How about instead of wasting time on this stuff, they build an actual
benchmark test for AGI. Something everyone in the community agrees needs to be
built, and basically nobody is working on. I suggest they start with the
"anytime intelligence test" from Hernandez-Oralloa as a stepping stone.

Oh wait, benchmarks aren't sexy. Nevermind.

~~~
gdb
> they build an actual benchmark test for AGI

This is actually the project I'm currently leading (goal #1 here:
[https://openai.com/blog/openai-technical-
goals/](https://openai.com/blog/openai-technical-goals/)). We should have
something interesting to share in a few months.

~~~
AndrewKemendo
Duh, forgot about the gym. Sorry.

~~~
gdb
We also have something new coming, which we hope will focus research and drive
forward progress towards AGI. (As always, looking for more engineers to work
on it. Just need a strong software generalist skillset. gdb@openai.com.)

