
DeepMind Challenges for StarCraft - arnsholt
http://gamasutra.com/blogs/BenWeber/20160314/267956/DeepMind_Challenges_for_StarCraft.php
======
scottjad
Starcraft is a game that is played with mouse and keyboard (or game controller
on console). I think in order to say an AI has beat a human, it should have to
use the same interface that a human uses. That's part of the definition of the
game.

In Go, anyone can place a stone as well as anyone else, it's deciding where to
place a stone that's the game. In Starcraft, not everyone can control the
interface to the game equally well. Playing Starcraft without using the same
interface as humans is like playing baseball with something other than a
baseball bat.

Also, I think it's unfortunate that they're choosing Starcraft 1: Brood War
(presumably because of API reasons) instead of Starcraft 2, since there are
many more top-level players playing sc2 right now to give it a run for its
money, with an evolving meta that could more likely adapt and challenge the AI
after it wins some matches. SC2 is where the tournaments and money is right
now (compared to BW), hence the best players devoting the most time.

~~~
barbs
From the 2015 AIIDE Starcraft AI competition report[0].

> ""Why not StarCraft 2?"

> This is the question we always get asked when we tell people we are doing a
> BroodWar AI competition. This competition relies completely on BWAPI as a
> programming interface to BroodWar. BWAPI was created by reverse engineering
> BroodWar and relies on reading and writing to the program memory space of
> BroodWar in order to read data and issue commands to the game. Since any
> program that does this can essentially be seen as a map hack or cheat
> engine, Blizzard has told us that they don't want us to do anything similar
> for StarCraft 2. In fact, most of the StarCraft 2 EULA specifically deals
> with not modifying the program in any way. We are happy that Blizzard have
> allowed us to continue holding tournaments using BWAPI, and they have also
> helped out by providing prizes to the AIIDE tournament, however until their
> policy changes we will not be able to do the same for StarCraft 2."

[0]
[http://webdocs.cs.ualberta.ca/~cdavid/starcraftaicomp/report...](http://webdocs.cs.ualberta.ca/~cdavid/starcraftaicomp/report2015.shtml)

~~~
beambot
So why not use Matt's version for SC2 that doesn't provide map hacks?

[https://graphics.stanford.edu/~mdfisher/GameAIs.html](https://graphics.stanford.edu/~mdfisher/GameAIs.html)

~~~
barbs
I imagine it also has something to do with convenience. Being such an old
game, the technical requirements are quite low, which allows them to easily
have that setup of multiple virtual machines on a few servers. Also, I believe
they can use the same CD Key for each instance, and it doesn't require a
connection to the internet, reducing variables and potential problems.

That said, it would be cool to see SC2 for this, and maybe we'll see it
eventually. But I think Blizzard would have to allow it in the end,
considering they've been in direct contact with them about this.

------
Monory
Well, anyone who says that advantage in micromanagement is not a big thing
should see "Automaton 2000" videos. Given that automaton is a map script and
thus can have virtually unlimited APM, it beats people from a horribly
disadvantageous positions (40 banelings vs 21 marine [1], or 100 zerglings vs
20 sieged tanks [2]).

[1]:
[https://www.youtube.com/watch?v=DXUOWXidcY0](https://www.youtube.com/watch?v=DXUOWXidcY0)
[2]:
[https://www.youtube.com/watch?v=IKVFZ28ybQs](https://www.youtube.com/watch?v=IKVFZ28ybQs)

~~~
nickpsecurity
I agree. It's one of only areas the AI's have been hood at. Pathfinding and
build orders being other two. AI should be limited to a specific number of
actions a minute comparable with opposing player or best human champ. Then,
we're grading it on its "thinking" instead of dexterity.

They better keep fog of war on too.

~~~
derefr
Or, the AI could be set up like AlphaGo was: as a computer _giving
instructions to_ a merely human player. I'd love to see what a powerful
StarCraft AI would do if trained to "predict and compensate for" not only its
opponent, but also the inevitable human errors in faithfully executing its
strategy. A certain Sector Command and Control Unit springs to mind...

~~~
dragontamer
A huge amount of Starcraft is the intense APM training regiment that the pros
undergo.

APM is a huge indicator of Starcraft "skill". Its not the only thing of
course, but winning "micro-battles" greatly changes the game.

I don't care how good Michael Jordan is. He'll never "win" a game of
basketball vs a standard professional by shouting commands to an average joe.

Similarly, no average joe will be able to perform a muta-harass while
retaining full-speed, taking only a single missile from a Turret. That sort of
"micro-skill" takes practice and dexterity.

~~~
derefr
You don't think it'd be interesting to take a player who's already one of the
top players, and measure the marginal gains of them doing their own
macro+micro, vs. just doing micro and leaving macro to the AI?

~~~
ktRolster
We can see that now, with the new Archon mode, which allows two players to
control the units together.

~~~
garrettgrimsley
I haven't been following SC2 since HOTS, are they really still trying to
achieve feature parity with Brood War?

[http://starcraft.wikia.com/wiki/Team_Melee](http://starcraft.wikia.com/wiki/Team_Melee)

~~~
ktRolster
_I haven 't been following SC2 since HOTS_

Too bad, it's really good.

------
dave_sullivan
> StarCraft is closed source, making it quite challenging to run simulations.
> One of the techniques used by AlphaGo is reinforcement learning, which
> involves a huge amount of simulation. In order for DeepMind to overcome this
> limitation, it’s likely that novel abstractions of the state-space will need
> to be developed.

For their atari work, deepmind was using raw pixel data from the screen. You
could do something similar for starcraft. I suspect the actual machine
learning question is 'given a sequence of n frames of the game, where should i
click next? repeat.'

A training set might look something like video of a game with a text file
recording X,Y positions for every click and a timestamp that can be synced to
the video. They need lots and lots of those.

From there things get drastically more complicated. To go from very
unstructured data like that and form higher level representations that can be
used for long term planning and strategy... That would be quite a feat.

~~~
dsl
Game replays (including all the fun data like player clicks, unit selection,
movements, and exact timing) are stored in an easily parsable format.
[https://github.com/GraylinKim/sc2reader/wiki/Replay-
Format-H...](https://github.com/GraylinKim/sc2reader/wiki/Replay-Format-Home)

~~~
HCIdivision17
This has the immense advantage that there's a large corpus of games available
to learn from as well. I don't _think_ Battle.Net saves all games ever (only
locally on the computer), there's still many thousands available online.

But man, if Battle.Net _did_ save all of them, there'd likely be hundreds of
thousands, if not millions, available. That'd be pretty spectacular as a
training set!

------
ktRolster
Artosis wrote his view on the topic, it's probably worth a read:
[http://espn.go.com/esports/story/_/id/15019778/infinite-
apm-...](http://espn.go.com/esports/story/_/id/15019778/infinite-apm-artosis-
deepmind-starcraft-part-2)

~~~
ShardPhoenix
It sounds like he has no idea about the machine learning basis of Alpha Go
(and therefore presumably any future Starcraft AI) and thinks all this complex
decision making has to be programmed in by hand (which would indeed be nigh-
impossible).

~~~
ktRolster
The interesting parts aren't his views of AI (since he doesn't have much
knowledge there) but rather his views on the complexities of the game. Because
on that topic he is an expert.

------
DanBlake
One of the issues to overcome is the advantage machines can have in individual
unit manipulations vs what a human can do in terms of APM

[https://www.youtube.com/watch?v=SZawUuAccnc](https://www.youtube.com/watch?v=SZawUuAccnc)

~~~
Mithaldu
The problem is, as ActionButton argues in this review of Diablo 2 [0],
Starcraft is a game that may contain some amount of strategy, but wherein the
element of strategy is strongly over-powered by a reflex-based combat system
more akin to Street Fighter than anything else.

[0] [http://www.actionbutton.net/?p=518](http://www.actionbutton.net/?p=518)

~~~
TulliusCicero
If that was true, then bots would already easily dominate humans, just like
they can for fighting games.

But they don't; even moderately competitive players can beat the best bots.

~~~
Umn44
"But they don't; even moderately competitive players can beat the best bots."

That's because the programmers are bad at AI for that kind of game. It just
means human beings don't understand how to make a good AI for a game like
starcraft. AKA the human being being the idiot that doesn't understand how to
make one.

------
confiscate
DeepMind is becoming a nightmare for South Koreans

------
proc0
I'm not as optimistic as most seem to be. I think SC adds a completely new
layer to the game, that none of the previously learned games had. It's not
enough to play the game just by rules that are implied by the pixels.

In chess, go, and even simple atari games, you have a clear way of reading the
current state of the game in which the next move will be based on. In SC, this
state space includes the predictions of your opponent. What will your opponent
build next? What will they try to do with their army? There's an element of
theory of mind here, where opponents try to project themselves and preempt
each other's moves.

For an AI to truly play and win against a SC pro, using only human level
interfaces, it would need to learn how human minds will be playing the game.
Otherwise, I think we can expect a strong AI with unconventional tactics that
SC pros will be able to fool with tricks and take advantage of because the AI
lacks a meta-game capability. A more "general intelligence" would be needed to
play at this level, not just a learning algorithm that learns from the pixels
on the screen.

------
hmate9
I think poker would also be a great challenge. It is a special game because as
well as the math, you also have to understand your opponent and adjust
accordingly. It'd be interesting to see weather DeepMind can beat professional
poker players who have different playing styles.

~~~
im3w1l
Wouldn't that depend on what you want to achieve? To maximize $/h you need to
understand your opponents weaknesses and exploit them. But if you just to
_beat_ the best humans, and don't care about the margin with which you beat
them, then playing a mathematically perfect game should be enough. And before
you ask, a mathematically perfect game can incorporate occasional bluffs, and
occasional calling of bluffs, this would be known as a Mixed Strategy.

Or am I missing something?

~~~
eru
That's about it for a two player game. Anything with more than two people has
more interesting / complicated properties.

~~~
im3w1l
One very real issue I'd imagine is that two players could cooperate out of
ignorance, i.e. either player could do better by deceiving the other, but they
are not good enough players to realize that.

------
nickhalfasleep
Does anybody else think teaching a rapidly evolving strong AI about how to be
really good at war might be a bad thing?

~~~
eru
StarCraft is not war.

~~~
uremog
Maybe they should go for Warcraft?

~~~
eru
I was tempted to write `StarCraft is not WarCraft.' in the first place, but I
managed to resist the pun. Seems like you've yielded.

------
nickpsecurity
Survey paper below introducing Starcrafy challenges plus all the AI's that
failed to conquer humans.

[https://hal.inria.fr/docs/00/87/10/01/PDF/survey.pdf](https://hal.inria.fr/docs/00/87/10/01/PDF/survey.pdf)

------
fatman13gg
It would also be interesting to see if DeepMind challenges Dota 2 one day
maybe. How they would handle a 5-man team game and adapting to major game
updates?

~~~
econnors
I'm more of a LoL guy, but i'd be excited to see this in action. I think it's
actually an easier challenge than starcraft - the action space is much smaller
and dodging/aiming AI already gives some players extreme advantages (against
the TOS of course).

~~~
huac
IMO the smaller action space/better defined win conditions is outweighed by
the greater playstyle diversity: SC has 3 races while League and Dota have
100+ champions to understand the interactions with. How is an AI supposed to
know that Tryndamere's ult has a 0.5 second cast time, but can't be
interrupted by any CC?

Personally, I think (all else being equal) Dota might be an easier starting
point than League because of its focus on item abilities, rather than champion
abilities. Anybody can use a BKB in Dota, but in League you have Fiora W, Morg
E, and a bunch of other things. You have a smaller set of abilities/effects to
worry about.

The dodge/aim AI can be situationally good (Cass bots) but still, the best
players do win against scripters. Scripters rarely make it into the elite
echelons of the ladder; there's a situation right now with a former pro Korean
player who started scripting but couldn't make it back into Master tier. You'd
expect a former pro with very in-depth game understanding to play at the very
tip top level with some 'aim assist' but it doesn't appear to be the case!

~~~
dolphingarden
This is incorrect as Starcraft has a much larger playstyle diversity than
League. Every league game follows the same structure: with very high
probability, one player will be top, one middle, and two bottom, with one
jungler. Teams will farm with occasional gank attempts until the midgame,
where there is a slow transition from farming to objective taking and
teamfighting. This is relatively formulaic and easy to learn. In contrast,
from the start of the game in starcraft, players can choose from a wide
variety of strategies, ranging from early rush strategies (with low and high-
tech variants), harass-and-expand strategies, greedy expand strategies, and
safe expand strategies. Also, it must also construct a prior over strategies
for its opponent that changes by map, as map properties change the viability
of certain strategies.

Your example of script-users not attaining high ranking in online play for
league of legends is also misguided. Because the action space in league is so
much smaller than in starcraft, it is much easier to develop an AI with game
strategy competency for league. Combine that AI with perfect mechanics, and
human players should be falling very readily.

~~~
huac
> Every league game follows the same structure: with very high probability,
> one player will be top, one middle, and two bottom, with one jungler.

The prevalence of lane swaps (2 go top, 1 goes bottom) in competitive is one
major counterexample. In fact, these often turn into having 3 or 4 people in
one lane at the start of the game, forcing a fast push strategy.

> Teams will farm with occasional gank attempts until the midgame, where there
> is a slow transition from farming to objective taking and teamfighting.

Stereotypical Chinese LoL matches are nearly all teamfights, for better or for
worse. Regardless, laning and midgame is where I think you should consider my
point of champion diversity as well. Here, mistakes can be more readily
mechanically exploited (late game, mistakes are punished strategically), e.g.
if a laner is far from baron without TP, exploiting a 5v4 fight is a strategic
decision, whereas missing certain skills early on in certain matchups means a
severely negative outcome if the opposite laner has the mechanics, e.g. no
flash and Q on CD as Lux means any Morgana can flash ult for guaranteed kill.
These have a bigger impact than racial considerations and probably make up for
map considerations (idk)

The script user example is one of mechanics: top players still win against
those with pixel perfect scripted mechanics. This is despite having some
scripters having 'game strategy competency' (that might be a real thing, but I
interpret it as a player having real game knowledge).

IMO the biggest challenge to a human player playing against a team of 5 AI-
controlled players is that of communication; perfect understanding of each
others' intent is a far cry from the communication of even practiced teams.

------
hiq
An AI with human-like abilities (so no infinite APM for instance) would be
great for balance problems in competitive video games. Just run the AI a
certain number of times, and if some race or unit always wins the game,
there's definitely a problem.

------
klog
I would say Koreans are on supercomputer AI level in this game.

~~~
broodbucket
StarCraft would indeed be interesting for one of these challenges, however the
peak of human skill in this game has already been lost. Tournaments are
dwindling, many players have moved on to other games like StarCraft II or
retired. The remaining players aren't at the level that they were, say, 5
years ago, when there were two major televised starleagues.

Even if some AI does defeat Flash in 2017, would they have defeated Flash in
2010?

The mechanical skill required for this game means you simply can't come back
to the game after a year and resume where you left off. Knowledge and
understanding aren't enough, your fingers must be lightning.

------
stared
When it comes builds: [https://www.rockpapershotgun.com/2010/11/02/genetic-
algorith...](https://www.rockpapershotgun.com/2010/11/02/genetic-algorithms-
find-build-order-from-hell/)

For micro, there are scripts:
[https://news.ycombinator.com/item?id=11325855](https://news.ycombinator.com/item?id=11325855)

So, while it may be not easy to conquer humans, I don't see why it is out of
our current reach.

------
timwaagh
I'm rather suprised there are no player beating ais for starcraft. i dont
think it should be very hard as decisions need to be made very quickly in this
game and humans are bound to make more than a few mistakes. but perhaps i'm
underestimating the problem. it may have something to do with the limitations
of the api though.

~~~
ygra
You're underestimating the problem. And there are several:

\- Controlling individual units: scouting, resource gathering, micro-
management while attacking/retreating, ...

\- Resource management and planning: knowing when and where to expand, save
resources for certain points in the tech tree, e.g. so you can immediately
build a bunch of units once a crucial research or building is complete, ...

\- Figuring out what the opponent is building and what he could be doing:
where is he and what race, what tech buildings are there, when does he start
to harvest gas, key tech buildings may be hidden in other places of the map,
essentially even _not_ being attacked with certain units at a certain time may
be a clue to some strategy

\- Reading and using the terrain properly: High ground has an advantage,
Terran can wall off certain entrances with a depot and a barracks, where would
be good places for drops (and with what units), figuring out where the enemy
could come from and which places to defend – all those vary by map quite a bit

\- Finding a good strategy to counter what the opponent is doing: This ties
into a few of the things above, because first it must be known what the enemy
is doing. Keeping the strategy flexible enough to counter the enemy's
counters, etc.

Some of those are necessary to solve in real-time (unit movement), some of
those can be run in background and don't need to be frame-exact (strategy,
etc.), which helps a bit. But there are still quite a few parts to playing the
game, all of which are interconnected and require information from other
parts. Many successful bots currently are rush bots, which exploit that most
bots are weak in the beginning and are easy to create. Others rely on a number
of fixed strategies and build orders that are chosen based on what the enemy
is observed to do.

------
forrestthewoods
I want to see StarCraft next and then League of Legends/DOTA. The control
advantage an AI has in StarCraft is immense. It exists in League of Legends,
especially when perfectly coordinating 5 different characters, but it's less
impactful.

~~~
bottled_poe
The control required to master SC2 is far beyond any MOBA style game. AI for
MOBA games seems trivial compared to AI for a large scale RTS. It's the
difference between controlling 1 unit vs 100 units.

~~~
forrestthewoods
I don't think I agree with that at all. If you can control 1 unit it's trivial
to control 100.

I might argue that the AI vs AI challenge in Starcraft is deeper and more
difficult to master. But for AI vs Player the Starcraft AI has an enormous
mechanical advantage. Being able to harass mineral lines in four different
places is a big deal. Being able to perfectly micro multiple lines of
attack/defense is a big deal. It's very unfair and very much in the AI's
favor.

Chess < Go < Starcraft < League of Legends

In my moderately educated opinion.

~~~
komaromy
> If you can control 1 unit it's trivial to control 100.

Control them, sure-kiting 100 units at a time would be a breeze-but using them
together in a coherent way is drastically more difficult.

~~~
lumpypua
From an above poster:
[https://www.youtube.com/watch?v=IKVFZ28ybQs](https://www.youtube.com/watch?v=IKVFZ28ybQs)

~~~
bottled_poe
The developer of the AI in this video even states that this AI is applies 300
APM _per unit_. That's 30000 actions per minute. There are simply not enough
decisions to make in League of legends to make use of that power.

------
jmnicolas
IMHO it's not artificial intelligence it's just intelligent scripting. A real
AI, like a human shouldn't be programmed for something specific but adapt
itself to what it encounters.

------
d0m
I know bliz doesn't want it, but it would be so much better with sc2.

------
ucho
Maybe Civ? I guess the single game takes too long but I still would love to
see it as default AI is totally crap and relies on handicaps.

------
ddxv
This is an amazing project. Thank you for that post. I would love to find some
teams working on this who are interested in adding junior level engineers.

------
anaconda123
damn bored

