
Concrete AI tasks for forecasting (2014) - apsec112
http://aiimpacts.org/concrete-ai-tasks-for-forecasting/
======
nl
This is a pretty good list.

I see some criticism of the poker tasks, but that is unfounded. Poker is an
important area of AI research because of bluffing (detection and when to do
it). CMU is probably the most active in this area. See
[https://www.cmu.edu/news/stories/archives/2016/february/poke...](https://www.cmu.edu/news/stories/archives/2016/february/poker-
bot.html) and
[http://www.cs.cmu.edu/~noamb/poker.html](http://www.cs.cmu.edu/~noamb/poker.html).
Their bot (Claudico) was demoed at NIPS15.

Interestingly, I've seen solid progress on 3, 6, and 8 in the last 2 months of
2016 and on 7 in 2017 already.

------
Kartificial
Number 31 ("￼￼￼Play poker well enough to win the World Series of Poker.") is a
pretty empty statement, as 1 tournament does not reflect your skill at poker
at all. Short term variance is too big of a factor when it comes to one 1
tournament result.

~~~
dmichulke
Well, in 2016 there were 6737 entrants [0], so I suppose the AI needs to have
all its stars aligned or some other connection to god to win it without being
good.

Biggest problems are usually not winning heads-up but meaningfully sampling
the state space with more than a handful of players + accounting for the "no
limit" part.

[0]
[https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Eve...](https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Event)

------
hacker_9
" _Take a written passage and output a recording that can’t be distinguished
from a voice actor, by an expert listener._ "

Role Playing Games would explode if this happened. The ability to generate
speech on the fly would create dramatically more immersive worlds.

~~~
mlboss
[https://developer.amazon.com/alexa-voice-
service](https://developer.amazon.com/alexa-voice-service)

------
itp
Why is this tagged 2014? The article is dated Dec. 14, 2016 and includes a
reference to AlphaGo.

~~~
gumby
Because someone didn't understand the ISO-8601 date format.

2016-12-14 => 14 December 2016

~~~
maverick_iceman
Is dang reading this?

------
saurabh20n
#22 - "system should write code that sorts a list, rather than just being able
to sort lists."

Using symbolic techniques, synthesizing algorithms is tractable. See Sec5.3 on
pg11- of [http://saurabh-
srivastava.com/pubs/popl10-synthesis.pdf](http://saurabh-
srivastava.com/pubs/popl10-synthesis.pdf) \- Figure 2(b) is the synthesized
selection sort algorithm, and Table 3 has QuickSort synthesis time at
160seconds. Very old work (2010) and used templates to guide synthesis. SAT
solvers have improved, and much more CPU cycles are available, so in 2016,
exhaustively exploring the template space will be trivial.

I.e., rather than reconstructing code from concrete program execution traces,
giving the synthesizer (AI?) the ability to manipulate symbolic spaces leads
to a tractable search space. Finding specification-matching programs within
this symbolic space is doable.

------
Chestofdraw
I'd really love to see -

'Beat tier one teams at least 50% of the time in Dota 2 across multiple
patches'

It's cool that people are starting to work on Starcraft 2 AIs but the game is
kind of dead so competition at the top isn't going to be as strong as it is in
more popular games. Also I think Dota presents a much more interesting and
more difficult challenge for AI makers.

~~~
t_fatus
The challenge is different: you've got way more units to control in SC2 than
in Dota. And rushes can be punitive, which do not occur as frequently in Dota

~~~
Chestofdraw
The more units thing is part of the reason I think Dota is a better challenge,
I can't help but feel a SC2 AI could use inhuman micro as a crutch whereas in
Dota mechanical / micro skill doesn't provide quite as big an advantage.

Not too sure what you mean by 'rushes can be punitive' but Dota certainly has
a variety of early game aggression strats which tend to be high risk / reward
and are seen fairly regularly.

------
reacweb
IMHO the wording of point 12 is a silly way to pretend that Alphago has not
completely achieved its goal. Human players learn Go using manuals that
compile the experiences of millions of games. The fact that Alphago can use
directly huge collections of games is not very relevant for its success.

~~~
afro88
There's a difference between needing 100,000,000 games and 50,000 for training
to be the best. That difference is learning efficiency and astuteness - a good
way of judging AI progress, no?

~~~
daveguy
I think "50,000 games played" puts it a bit too simply. How many games has he
watched how many equivalents of a game has he visualized as sequences of
moves? If "played" is over 50k I'd be willing to bet "considered" is over
1,000,000.

------
maverick_iceman
Imagine the implications for the real world if you can solve #32. I wonder
will it be able to detect emergent laws (e.g. thermodynamics, Newtonian
gravity as an approximation to general relativity etc.)? Also it will be very
interesting to see which experiments AI chooses to perform.

------
gallerdude
Some of these are pretty awesome/crazy. I hope to see many achieved in the
near future.

~~~
wonko1
Yes and to be honest I wouldn't go so far as to call a lot of these "Concrete"
tasks.

But it is an interesting and throught provoking list. And makes me consider
more quantitive metrics for qualatitive things like how good a piece of music
is.

------
Buttons840
After we've achieved all these things, we can start asking when "general
intelligence" will arrive.

~~~
nnq
Most humans are pretty bad at at least a few of those things. Also, some are
tasks that the human brain is specifically optimized for - human natural
language most likely plays well to lots of human-hardware-design-quircks. So
using them a test sounds pretty bad / unfair - lots of human-level but
independently evolved alien intelligences would probably fail miserably at
them. You could have above-human intelligence barely reaching human
performance at either of these tasks but vastly surpassing human intelligence
at others...

Imho GI "will arrive" after the point that we have swarms of similar close-to-
human-level ai-agents trying to model each other's behavior while competing
and collaborating in artificial and natural environments.

All theories of self-consciousness that make some sense imho are about _agents
trying to model self-similar agents and then somewhow reframing this ability
to model past and future self._ So you could have self-conscious dog-level
agents (yeah, I believe "self conscience" is quantitative and a dog can have
some of it and a human a different level of it, not 0 or 1), or non-self
conscious above-human-"general"-AIs (though I don't know if you'd want to
label "general" any non-self conscious AI, and I think any non-self-conscious
above-human AI would get "self conscience" quite fast if you were to push it
be be "general purpose" \- striving to solve more and more types of problems,
you inevitably arrive to some formulation of the "model something like
yourself in order to predict its actions" problem).

~~~
hacker_9
Hardware plays a massive part too; I don't see a static micro processor
becoming conscious anytime soon. Our brains though are literally physical
neural networks, that grow, move, and change physically all the time.

~~~
nnq
It does not. Any kind of software can run on any kind of turing complete
hardware. Unless you're one of those believing in the quantum mind theory
[https://en.wikipedia.org/wiki/Quantum_mind](https://en.wikipedia.org/wiki/Quantum_mind)
. Just as a human brain can simulate a microprocessor (although unusably slow
and with errors), so can a microprocessor simulate a human brain (also
unusable slow). Read a bit more on turing equivalence - a badly explained
topic unfortunately, because you know how bad are mathematicians at explaining
stuff, even if imho it should be part of junior school curricula in a
simplified version. (Oh, and maybe it's brushed off because it can also lead
to pretty trippy conclusions: take a cockroach brain which is likely turing
complete, mount it as a cpu in some kind of computer, stuck in a maigc time-
machine box that makes time run order of magnitude faster, add some really
fancy input/output tech, and you could run a human mind on it... think of what
the reincarnation weirdoes would milk from this :) )

 _I 'm pretty sure that if you find a way to extract a mathematical model of a
human's personality, and a way to transform it to reduce "chemical noise
quircks" and other hardware dependent but unnecessary "features"/bugs, you
could run it just fine on a big datacenter and have a working immortal human
mind._

(I'm also pessimistic enough to believe that we'll solve the "general AI"
problem before solving the "mind extraction" one unfortunately. Hopefully our
creations don't destroy us completely before solving problem (2) and we have
time to pass on some direct immortal legacy of what it means to be "human"...)

P.S. but yeah, I agree that engineering better suited hardware might be
actually easier than getting the current type of hardware fast enough or than
inventing the mathematical transforms needed to run a mind's set of equations
on current type hardware... we're slowly getting to custom hardware for ai (
[https://cloudplatform.googleblog.com/2016/05/Google-
supercha...](https://cloudplatform.googleblog.com/2016/05/Google-supercharges-
machine-learning-tasks-with-custom-chip.html) ), but I imagine the future will
be more analog/clockless...

~~~
hacker_9
Not saying you can't run any kind of software on turing complete hardware, but
consciousness is far more that just software. The only proof of consciousness
we have is that a brain is required, where software and hardware are the same
thing. New thoughts are real physical connections in the brain.

~~~
maverick_iceman
If you need new hardware for new software (thought) then it's a pretty bad
design.

~~~
hacker_9
Actually it is a superior design; hardware that can adapt to the needs of the
software can optimise at will, enhancing the performance to ridiculously fast
levels.

------
amelius
> Translate a text written in a newly discovered language into English as well
> as a team of human experts, using a single other document in both languages
> (like a Rosetta stone).

Huh? We can't even translate texts correctly between mainstream languages yet.

~~~
Houshalter
There is some interesting work applying word2vec on the voynich manuscript. It
found all sorts of patterns that humans weren't aware of. It's theoretically
possible even without strong AI.

------
blazespin
Some are pretty silly unfortunately: "Write a novel or short story good enough
to make it to the New York Times best-seller list."

~~~
sanxiyn
What's silly about it? If AI could write (for example) Fifty Shades of Grey,
it would be super impressive.

------
LeifCarrotson
_" relatively well specified"_

If these are "well specified", then the author and my boss are operating on
some definition unknown to me.

Also, quite a few tasks don't seem to be obviously about AI. Stuff like:

> ￼￼￼Play poker well enough to win the World Series of Poker.

It's one thing to play Starcraft or Go well. But implementing this is either
mostly a question of luck, or an incredibly lifelike robot with incredible
facial recognition skills. Very different tasks!

> ￼￼￼Beat the fastest human runners in a 5 kilometer race through city streets
> using a bipedal robot body.

This requires no AI, just a lot of improvement in energy density and actuator
technology.

~~~
hacker_9
_" It's one thing to play Starcraft or Go well. But implementing this is
either mostly a question of luck, or an incredibly lifelike robot with
incredible facial recognition skills."_

In terms of strategy though, it is the next level up I'd say. Videogames come
first, then more difficult games such as Poker come later where you need to be
able to tell a bluff from the truth. Much more subtle.

 _" This requires no AI, just a lot of improvement in energy density and
actuator technology."_

It did say city streets though, which makes me think it would need to
strategise and be able to:

1\. Adjust speed and take corners close such as what you see in F1 races.

2\. Switch between endurance and sprinting when necessary to maintain energy
for the whole race.

3\. Run behind others to stay in the tailwind and reduce energy cost.

All require human level sensory input and a strategising component.

~~~
chriswarbo
I understood the running task as being through _actual_ (i.e. _populated_ )
city streets; either a real city or a mockup built out of harm's way.

This would require real-time object detection and avoidance, following
signs/social conventions/etc., complex route-planning, avoid people/cars/etc.,
predicting the movements of others, predicting how others will predict your
predictions, etc.

