
StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer - mav3r1ck
https://arxiv.org/abs/1804.00810
======
calebh
For those who don't know Starcraft: microing refers to the small scale/precise
maneuvering of units during a battle, as well as activating special unit
abilities at certain times. In the lower level leagues microing is not as
important as having a good overall large-scale "macro" strategy. The microing
really starts coming into play at the higher levels where it can give the edge
needed to win a battle.

I am not very surprised that machine learning has been able to successfully
execute micromanagement. In my opinion the macro decisions are more
interesting, since they typically require higher level reasoning (and
comprehension of what the opponent is doing). We have yet to see an AI system
that can successfully execute macro strategy when playing against a human
opponent.

~~~
s_m_t
How can this be so? In my experience playing Starcraft the decision space for
macro is very, very, very limited. You are essentially locked into a handful
of viable build orders for the early game, a handful of timing attack
opportunities in the early and mid game, and after that things veer so off
track that very few players do more than wing it.

~~~
ReidZB
I'd disagree. At the pro level, choices like unit composition and expansion
timing are very strongly influenced by what the other player is doing. I've
seen videos where a caster (like Lowko) breaks down "the build order" that a
player did in a pro level game, only for the pro to turn around and say
something like: I wasn't following a strict build order, just reacting to what
I saw from the opponent.

I watched a pro game just the other day where one of the players noticed that
their protoss opponent had a late timing on their second pylon in their main
base - as in, it should have been there, but wasn't. That protoss player has a
well-known penchant for early game stargate harassment, so the other player
read this situation and reacted by delaying their first expansion and instead
producing early-game air defenses.

As another example, take unit composition. If a terran player is running a
bio-based army (marine, marauder, medivac), what sort of units should zerg
produce? Infestors are usually a good choice in this matchup. On the other
hand, mass infestors will be much worse against a primarily mech-based terran
army; instead you might see many vipers on the field (which, vice versa,
wouldn't be great against a bio army).

Pro players are essentially always scouting what their opponent's doing
because the macro decisions can make or break the game. The general sentiment
I've seen that is good macro skills will push you far up into SC2 leagues, and
it's not until diamond/masters that micro starts to really matter.

~~~
wesleytodd
I completely agree with your assessment.

I watch a fair bit of pro play, and also play enough to bounce between low
diamond and high platinum on NA. It is almost always my macro that wins or
looses games.

Most people at my level wouldn't even have consistent enough play to be able
to notice the difference between well and poorly micro'd games. But if they
miss an upgrade, or build the wrong comp, or worse, don't expand, that is the
win/loose for 98% of games below masters.

Which all around sucks for me, because my micro is on point :)

------
WhompingWindows
Creating a micromanagement AI is only the first step. This must be piped into
a macro AI, which produces the units with with micromanagement can be done.
Then, another critical layer beyond micro or macro is scouting the opponent
and predicting what their plan is. Next, you need to adapt your own strategy,
which must also be planned at the start of the match since you can't scout
instantaneously or perfectly. Finally, your own initial strategy choice must
be tailored to the map, to the player you're playing against, to recent
games/strategies other pros have exposed as powerful/weak. There are an
incredible number of layers of complexity to what a truly fierce Starcraft AI
would need to do.

And yet, something similar may have been said about Go, Chess, Checkers,
Poker, Jeopardy, or many other games before this. If enough talented minds and
powerful devices focus on Starcraft, how would humans compete? Does anyone
here think top pros would win in a best of 7 or a best of 11, for instance?
Which top pros, and BW or SC2? To me, I just think it'd be arrogant to assume
SC is different from all the other realms in which humans have been bested by
machines.

~~~
jon_richards
SC2 has a lot of things that help the player over an AI, like imperfect
information, a very large ruleset, and continuous movement and time.

The main advantage AI can exploit is "infinite micro" units. There are certain
units in SC2 that are primarily limited by someone's ability to pay attention
to them. Terran Reapers, for instance, in early patches could dance back and
forth just within range Zerg Roaches, but out of range of being
counterattacked (though the reapers would quickly die if player stopped paying
attention). This required such fast movement that it was almost never employed
except by professionals during the early game when not much was going on.

If an AI is not limited by the number of actions a person can take, these
"infinite micro" units become extremely powerful. In particular, most of these
types of units are within the Terran race and most powerful in the early game.
The best AI in the near future will likely focus on early all-in attacks using
these sorts of units (primarily reapers, widow mines, hellions, and medivacs).
The human player would have to rely on defensive structures that would put
them at a major disadvantage and then catch up in the mid-late game, relying
on the fact that the AI gets worse as the game goes on and becomes less
predictable.

Another thing to note is that Terran strategy relies quite heavily on killing
the enemy workers in the early game, but the inbuilt controls for targeting
workers are quite clunky and prioritise military targets. Players can target
individual workers manually, but generally do so with their whole army at
once, while an AI could select the perfect number of units required to 1-shot
a worker, give them the command, then select another group, and so on.

~~~
littlestymaar
> like imperfect information

The information is only imperfect for a human but an IA can have a way better
access to information: \- if you scout you opponent mineral line and click on
the minerals, you know exactly how much minerals he's harvested since the
beginning of the game. If you scout your opponent army and building, you know
exactly how much he spent. Then you can deduce if he has something hidden
somewhere or in production. Of course this is impossible to do for a human,
but really easy for a computer. \- scouting is easy but requires a lot of
actions, an IA could have one mutalisk wandering over the map the whole game
without being killed, which is impossible for a human unless you dedicate full
time on this.

Imho, beating humans to StarCraft isn't a big deal in itself, but it would
need a lot of work to build the AI to deal with the game's complexity. But
it's merely domain specific work that has little to no interest on AI in
general. Creating an AI that trains itself how to be good at StarCraft will
take much more times and effort, but at least it will provide value outside of
the specific problem.

~~~
vk23
> Then you can deduce if he has something hidden somewhere or in production.
> Of course this is impossible to do for a human, but really easy for a
> computer.

This is definitly not impossible for humans. High level SC players are able to
keep track of the mineral and gas spendings of their opponent in the early
game (After the early game there is usually no complete information to
calculate this). Just seeing the timing of taking the gas geysers gives you
information about the earliest possible time a certain unit/building can be
produced.

Of course a AI can scout more and take more variables into account but humans
are surprisingly good at this too.

> Imho, beating humans to StarCraft isn't a big deal in itself, but it would
> need a lot of work to build the AI to deal with the game's complexity. But
> it's merely domain specific work that has little to no interest on AI in
> general.

The same thing was said about early chess AIs and the Alpha GO project. In the
end both lead to major advancements in AI. If you solve the domain specific
problem starcraft you could easily apply the gained knowledge to similiar
problems.

~~~
jon_richards
At the top levels, it's a lot more about _where_ those units/buildings are
than whether they exist. You know they're going to be mining and spending
efficiently, so just seeing the number of workers gives you a very good idea
of how much other stuff they have. Pros also check to see what technologies
are being researched (buildings have a different animation while researching).

You can't get much more scouting than that because there is a cost associated
with scouting, either because you lose a worker's time or because you spread
out your army.

~~~
vk23
> You can't get much more scouting than that because there is a cost
> associated with scouting, either because you lose a worker's time or because
> you spread out your army.

I agree but I was mainly talking about the time it takes to perform the
scouting. A AI could scout 5 different places at the same time which is pretty
hard for a human to do without falling behind elsewhere. I know SC pros are
really good at multitasking but at some point the AI is just superior. If this
gives the AI a significant advantage is a different question.

------
daemonk
I want an AI assisted mode with API hooks where people can code their own
sophisticated "macros" that can be activated with a command/button. Limit the
code to X number of characters. It would be awesome to see a tournament with
this.

~~~
axxl
There is some stuff like this, you would see it on the ladder in SC2. Hacks
that would perfectly split your marines against incoming banelings (they
explode and do splash damage so the more you split the less effective they
were), or auto burrowing/unburrowing roaches (which hides them from the enemy
and they heal underground). It sounds cool but it’s way cooler when you saw a
pro doing all that stuff themselves almost as good and still going back and
building units etc.

~~~
daemonk
I appreciate the manual dexterity, but imagine the amount of strategy that AI
assisted play would open up.

~~~
TulliusCicero
I agree with d1zzy, if you don't like the micro part of Starcraft, there are
plenty of strategy games that cater to a more strategy-focused style of play.

~~~
daemonk
I like the micro just as much as anyone else. But I don't agree that there is
just one way to play a game and that if someone wants to deviate from it,
somehow that goes against the purity of the established way. Blizzard has
benefitted greatly in the past by letting the public mod their games (dota). I
don't see anything wrong with extending that to Star Craft.

------
taylorswift_
Interesting, outside of the academic merits of the research, I wonder if they
could partner with Blizzard (or other similar RTS game studios) and start
shipping games with AI that is very difficult to beat? That would usher in a
new era of solo and offline game play that could be very fun and challenging!
Plus we'd be training Skynet, err, I mean the neural network in the process!

~~~
AmericanChopper
A 'perfectly' micro-ing AI is both boring to implement and boring to play
against. Somebody made one for SC2 back when that was popular, take a look at
these videos to see how uninteresting it would become after the first attempt.

[https://www.youtube.com/watch?v=0EYH-
csTttw](https://www.youtube.com/watch?v=0EYH-csTttw)

[https://www.youtube.com/watch?v=IKVFZ28ybQs](https://www.youtube.com/watch?v=IKVFZ28ybQs)

[https://www.youtube.com/watch?v=DXUOWXidcY0](https://www.youtube.com/watch?v=DXUOWXidcY0)

~~~
make3
yeah but I'm pretty sure these bots still get crushed by humans because of
macro, planning and positioning, which is why it takes top firms on the planet
so long to solve the problem

~~~
AmericanChopper
If the AI wasn't very smart, then you'd expect there to be some optimal path
that would beat them. But Starcraft has always been a game with very little
skill ceiling on micro. Well micro'd units have the potential overcome huge
macro advantages. A famous real-world example would be Boxer's Immortal
Marines:

[https://www.youtube.com/watch?v=WJp0t9n8DWk](https://www.youtube.com/watch?v=WJp0t9n8DWk)

------
taurath
I've always felt like macro is pretty easy to automate, once you've identified
a lot of the potentials in the early game you can get going pretty well. Is AI
playing against each other getting to some sort of endgame consistently
(against equal opponents), or are they having blowouts because of some small
difference in midgame?

Also wondering what the interface is - are they creating a virtual
mouse/keyboard to enter inputs, are they playing in some sort of headless
mode? How do they deal with recognizing units?

~~~
loser777
If this project uses the BWAPI (which it says it does in the paper), then I
assume that this doesn't need to recognize units as that state is available
from the game. On the other hand, the DeepMind "research environment"
([https://deepmind.com/blog/deepmind-and-blizzard-open-
starcra...](https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-
ai-research-environment/)), presents a simplified view of the map that
requires AIs to recognize units as simplified blobs on the map and issue
commands that more closely resemble what a human would do via mouse and
keyboard.

------
wesleytodd
But can your AI learn to drop mules so it can capture some escaping ultras?

Better link:
[https://clips.twitch.tv/BusyDrabFennelRedCoat](https://clips.twitch.tv/BusyDrabFennelRedCoat)

~~~
wesleytodd
Funny enough, after I posted this as a joke I realized this is exactly the
kind of thing an AI would be incredibly good at finding. It makes it all that
much more impressive seeing Buyn do it.

------
mLuby
Three great short videos showing an old StarCraft bot in action. I recommend
watching #3 then #2. The second shows a pro player for comparison.

[https://www.youtube.com/channel/UCOBfAiPhRlKOhdJHaAJ2OpQ](https://www.youtube.com/channel/UCOBfAiPhRlKOhdJHaAJ2OpQ)

------
partycoder
The thing with StarCraft is that there are many types of units and types of
damage, you need to alternate between moving and attacking, different units
have different roles in a formation, etc.

When I pick targets I prefer to kill units that deal the most damage or have
low HP first.

------
spectaclepiece
Anybody know if the military in any country is looking into using similar
strategies to manage real wars?

