
Tough times on the road to Starcraft - phenylene
http://www.codeofhonor.com/blog/tough-times-on-the-road-to-starcraft
======
reitzensteinm
The part about hand maintained lists is a common pitfall in game development.
When you're trying to push out frames at 60fps, you generally need to be
caching quite a lot of values, for example, "Get all units that are moving" or
"Get all units from side x".

The absolute best way to handle this is to start off with safe, conservative
functions, that don't cache at all - create a list, iterate over each entity,
add whatever fits the condition.

Then, after profiling to see what's slow, you hand roll a system that keeps a
specific list updated continuously; and then importantly, keep around the old
function, and in debug mode compare the lists occasionally (I usually do it
every 100th access, so the game is still playable).

After that, you may not know what's causing a cached value to go out of sync,
but you can at least know it's happening, instead of getting some of the most
convoluted and maddening bugs imaginable. The kind that make you want to take
up farming (and I'd be a shitty farmer).

~~~
marshray
_The part about hand maintained lists is a common pitfall in game
development._

Whenever I hear something about hand-maintained data structures I think
probably some manager or lead programmer banned C++ templates and other
abstraction mechanisms because they were "too tricky and hard to understand".

~~~
Peaker
Standard templated linked lists are suboptimal. Simple C lists ala Linux's
list.h are simpler and more optimal, behavior-wise.

~~~
marshray
Well I didn't say you had to use std::list. One can implement exactly the same
behavior as Linux list.h in C++, but type-safe and namespace-scoped.

But some managers prefer #defines. To each his own.

~~~
Peaker
The behavior of list.h essentially boils to downcasting from a list node to
its container. Allowing multiple list nodes in the same container, and re-use
of the same node for different lists. How do templates make this type-safe?

~~~
marshray
[http://www.boost.org/doc/libs/1_51_0/doc/html/intrusive/list...](http://www.boost.org/doc/libs/1_51_0/doc/html/intrusive/list.html)

 _How do templates make this type-safe?_

By using composition instead of inheritance, it avoids the downcasting.

If you want to, you can even have objects that are elements in several
different lists using tag types to let the compiler keep the lists from
getting cross-linked. Or you may prefer to use the node pointers for different
things at your discretion.

Self-removal on destruction can be useful too. All without any more run-time
overhead than the two pointers you need in C.

~~~
Peaker
I tried reading the code in intrusive/list_hook.hpp and intrusive/list.hpp,
and it's quite difficult to read. However, it does seem that it inserts an
extra pointer to the member (beyond the prev/next) in order to avoid the
"cast". If it indeed does so, it is again less optimal than Linux's list.h. On
64-bit machines, 8 extra bytes for each list element is not always negligible.

Also, the examples do not showcase using the same list member node to put in
multiple lists. There's also the unsafety of putting the list node in a union
(mentioned in the article) which adds unsafety regardless of templates or not.

I agree about destructors improving safety, I was specifically talking about
templates.

With list.h, it is possible to get the same safety (unless I am
misunderstanding something about the boost example) by wrapping the list
anchor and node structs with a struct (typically done by a macro to define the
anchor/node), which adds an array-of-0-elements of the type of the container.
This allows adding the same type checks demonstrated here using templates,
implemented in the list iteration, insertion/deletion and downcasting macros.

~~~
marshray
_I tried reading the code in intrusive/list_hook.hpp and intrusive/list.hpp,
and it's quite difficult to read._

Yeah those Boost projects sometimes go off the deep end with the
overengineering and abstraction a bit.

 _However, it does seem that it inserts an extra pointer to the member (beyond
the prev/next) in order to avoid the "cast"._

If it's a true C++ pointer-to-member often those can be completely optimized
out if the compiler has the type information available at the time.

I've done a (much simpler) intrusive DLL template that certainly did not have
that additional pointer, but I'm not sure if it met all of your requirements.
I know it supported membership in multiple lists, but I don't think they were
heterogeneous collections.

 _If it indeed does so, it is again less optimal than Linux's list.h. On
64-bit machines, 8 extra bytes for each list element is not always
negligible._

Agree, I'm all for performance. At least when I'm on that kind of project.

 _Also, the examples do not showcase using the same list member node to put in
multiple lists._

For that you could derive from multiple base hooks with different tag classes,
or compose multiple member_hook<class T, class Hook, Hook T::* PtrToMember>.
Each would have a different PtrToMember, but it would be known at compile time
and thus should be optimize-outable.

 _There's also the unsafety of putting the list node in a union (mentioned in
the article) which adds unsafety regardless of templates or not._

Boost has a safe type-discriminated union class you could probably use for
that if you wanted to.

 _With list.h, it is possible to get the same safety (unless I am
misunderstanding something about the boost example) by wrapping the list
anchor and node structs with a struct (typically done by a macro to define the
anchor/node), which adds an array-of-0-elements of the type of the container.
This allows adding the same type checks demonstrated here using templates,
implemented in the list iteration, insertion/deletion and downcasting macros._

Perhaps you could add an array of size 1 to represent the element data
following the last pair of pointers. Or I may not be understanding this. Can
you point me to an example online?

------
steve8918
I acknowledge Starcraft's dominance, but my personal favorite game of all time
was Total Annihilation. At the time (98/99), it was probably the most advanced
game out there. I played my cousin across the country over 33kps modem, and we
each had 500 units moving around at the same time. With 1000 units total, each
3D rendered, the action slowed down a lot, but it never crashed, and allowed
us to play entire games across the country reliably. It was truly a feat of
software engineering.

Chris Taylor (the creator of TA) came out with Supreme Commander in 2007,
which I bought a new computer for just to play, but I only played it for a few
months, more because I got busy rather than not enjoying the game. But both
games never seemed to catch on as much as Starcraft 1 and 2 have. I'm
seriously considering picking up Starcraft 2 but it's already 2 years old, so
I think I probably missed the boat.

~~~
ericdykstra
...And I was the guy that played the turned based Lords of the Realm II for
some reason.

Anyway, Starcraft 2 is still pretty active on the casual side, and with the
new expansion coming out later this year (already in public, but not open,
beta) it will surely drive a flock of new players. On the note of Supreme
Commander, one of the top players of that game has been a Starcraft 2 pro
since beta (TheLittleOne, aka TLO).

~~~
ricardobeat
Hey, I played - along with my father - LOTR II too, it was nice :) I still
have some memory of the soundtrack and effects, maybe the best part of that
title.

I still miss something like Commandos: Behind Enemy Lines, that was a
surprisingly great game.

------
jimrandomh
The main takeaway I got from this article is don't mix data structures and
application logic. This matches my experience; complex data structures work
fine if all the operations performed on them are in one place where they can
be unit tested (or at least carefully inspected), but it works very badly if
they're scattered or mixed with other things.

------
soup10
Are AAA games still coded like this? Where you hack your way to the finish
line. Or is there more process and things like code reviews?

~~~
chipsy
The 90s represented a period of growing pains where the industry was scaling
its budgets and team sizes very quickly and, for the most part, didn't have
any processes in place to deal with scale or to maintain a codebase for the
long term - why would they, when up until that point, games usually shipped
within a year or less, and the technology changed with almost every project?

Today the picture is still varied, but engineering practices have generally
improved - for example, daily standups are popular with many studios now,
there's a corpus of books and lectures about how to architect an engine or
various subsystems, etc. A lot of today's commonplace material was being put
into production for the first time then, and was previously only known in some
other context - research, high-end graphics and simulation, etc.

In current AAA the bottlenecks mostly lie with the art and design teams.
Programmers still have plenty to do, but they're also quite frequently
building on old technology that has some roots in the 90s - which entails a
whole different set of problems.

------
deathhand
I never knew the importance of Storm.dll but now it makes sense on why it was
always corrupted.

------
latimer
Really interesting read. He mentioned that he'd talk more about the pathing in
another post. I hope it will explain something I've always wondered about
which is why the dragoon pathing/AI was not so good compared to other units.

~~~
nialo
I think the explanation is actually pretty simple: because of the combination
of how they had to break the tiles up into smaller tiles and the dragoon's
relatively large size. The AI imagines it can get through spaces it can't, or
a gap turns out to be filled with other units, or similar.

(disclaimer: I'm just guessing, no inside knowledge. certainly feels right
though)

~~~
netcoyote
[note: I wrote the StarCraft article]

You're right -- it was related to the size of the units. Because they were
larger they needed to find wider paths that weren't obstructed by terrain or
other game units.

~~~
j2bax
Thanks for the countless hours of SC fun over the years. Amazing how far
things have come, but really how little has actually changed when it comes to
what makes the game fun and engrossing.

Did you guys have any idea what sort of an impact it would have in Asia (more
specifically with the Korean players)?

~~~
netcoyote
Nope. We thought we would sell about 4000 copies in Korea, to the extent that
anyone on the team was thinking about it at all. Timing is everything - we
just happened to hit the Korean market when everyone lost their jobs and
started playing in PC Bang. Kinda lucked out and sold a copy to every man,
woman and child in Korea.

------
kennywinker
I wish to know more about the "voice-to-phoneme" compressor he mentions.
Anyone have any links handy? Google is failing me.

~~~
polemic
Yeah, how cool would that have been. If you could do a JS port, you could you
use that to build a super lightweight Skype clone :D (audio capture from
Flash, or Mozillas audio capture..)?

~~~
grimboy
<http://en.wikipedia.org/wiki/WebRTC>

------
jmtame
One thing I always think about is how the code works behind games like
Starcraft. I find myself trying to understand the data structures and
algorithms behind seemingly simple mechanisms like creep spreading or
revealing the fog of war. Are there any other writings like this that talk
about the technical side of popular games?

~~~
cpeterso
Here are some game tech articles that I found interesting:

* Fixing Pathfinding Once and For All: <http://www.ai-blog.net/archives/000152.html>

* Scenegraphs: Past, Present, and Future: [http://www.realityprime.com/articles/scenegraphs-past-presen...](http://www.realityprime.com/articles/scenegraphs-past-present-and-future)

* Evolve Your Hierarchy: Refactoring Game Entities with Components: [http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy...](http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy/)

~~~
Too
Evolve your hierarchy is a really good read for everybody, not only game
programmers. It shows in a very convincing way why the classic Cat : Animal
inheritance model taught in every Java-school doesn't work when things scale
up and proposes a better alternative.

~~~
cpeterso
Reading this Starcraft story and "Evolve Your Hierarchy" reminds me that, in
retrospect, most of the class inheritance hierarchies in my student and early
professional projects were bad design. These days, I think implementation
inheritance is a code smell.

~~~
Evbn
"Prefer composition over inheritance" is an item in Bloch's Effective Java.

------
tsunamifury
Interesting to see how you feel bad about shortcuts you make at the beginning
of a project to save time, and later regret them wishing you had done it
'right' to begin with. However the cost of doing it right at the beginning
sometimes is greater than the difficulty of hacking to the finish line at the
end.

~~~
chrischen
There is no finish line.

~~~
nostrademons
Yeah, the problem is that software projects always take 10x as long as you
expect them to, so when you think you are 90% of the way there, it's more like
you're 9%. If you _know_ that you are only 9% of the way there, you will
develop in a very different way than if you think you are 90% there,
preferring long-term code health over quick hacks. (This in turn might shorten
the total project so that you're 20% of the way there, but you can't count on
that, because if you start assuming it you'll get careless and lose the
benefit of long-term thinking.)

~~~
jdlshore
Although I agree with your sentiment, I read the parent as saying something
different. There's literally _no_ finish line. It's not just that projects
take longer than planned.

With rare exceptions, software _begins_ when it's shipped. What everybody
thinks of as the finish line is actually the starting line. Successful
software is maintained, extended, and enhanced for years and even decades
afterwards.

~~~
eru
> With rare exceptions, software begins when it's shipped. What everybody
> thinks of as the finish line is actually the starting line. Successful
> software is maintained, extended, and enhanced for years and even decades
> afterwards.

And games often are among the rare exceptions.

~~~
flatline3
Actually, not so much. Leveraging the technology from your last game
development cycle is par for the course.

~~~
ANTSANTS
You're missing his point: Large or small parts of a codebase may be reused
from project to project, but, especially in the days of Starcraft, games are
often released as finished products, closed cases that the developers quickly
move on from after shipment.

Digital distribution has made the subscription-based (MMO of your choice),
"MVP"/"paid beta" (Minecraft, Terraria, and other indie games), and long term
microcontent/paid update (TF2, various iOS games) approaches more viable than
ever, but many game genres and corporate cultures are very much locked into
the idea of shipping a finished product and moving on from it.

As an example of both the modern "continuous updating" and traditional/greedy
"shipment oriented" approaches to development in a genre traditionally
considered "ship and forget", you could imagine a company that makes adventure
games (or interactive fiction or visual novels or whatever you want to call
them) for smart phones that reuses and improves their game engine from release
to release. Since a well-done adventure game _should_ still be appealing even
years after its original release, one approach would be to backport engine
improvements to their older games in order to make them more attractive to
long-tail customers. You might think of this as "we're selling stories, not
engines." A more traditional approach (still employed by many Japanese
developers) is to wait for a significant hardware or platform update and re-
release the game, usually (but not always) updating the visuals or enhancing
the interface, often several times, in hopes of milking fans dry with minimal
effort (lazy ports) or risk (updated rereleases of games they already know
people will buy). With the success of iOS and Android as continuously
improving platforms in contrast to traditional static consoles, we may see
less of the latter strategy in coming years.

On the other hand, Activision's Call of Duty series has been very successful
sticking to an unquestionably franchise-oriented approach. Each game is
basically the same as every other one before it, dressed up in a slightly
different coat of paint, with slightly tweaked gameplay mechanics, etc, yet
people enthusiastically plop down $60+ for every new release. Activision has
no incentive to backport their engine improvements to their older titles
because they want people to move on to the latest and greatest installment,
and no one wants to buy the old installments anyways -- no one actually wants
Modern Warfare 1, 2, or even the latest installment of today, specifically.
They just want access to the latest iteration of the Call of Duty formula and
all of its players, and they're willing to pay $60+ every year to do so.

------
suresk
Interesting read. Reading articles like this makes me wish the sources to some
of those older RTSes (Starcraft, Warcraft, Age of Empires) were available to
look at - I'm sure there'd be some interesting stuff in there.

~~~
grimboy
Looks like this is as close as you'll get for now:
[http://bazaar.launchpad.net/~stratagus/stratagus/trunk/files...](http://bazaar.launchpad.net/~stratagus/stratagus/trunk/files/head:/src/)

ETA: Just remembered. They open source 7 Kingdoms a few years back so you can
find out what the code from a commercial RTS looks like.
<http://www.7kfans.com/wiki/index.php/Download>

~~~
rurounijones
MechCommander 2 source also available: [http://www.microsoft.com/en-
gb/download/details.aspx?id=1145...](http://www.microsoft.com/en-
gb/download/details.aspx?id=11457)

------
wavesounds
I really like this article. I think the dynamic involved in working tons of
hours on a project like this is really interesting. On one hand you have peer
pressure and pride in your work. And on the other you have code degradation
with lack of sleep. I wonder if he feels becoming a VP of Blizzard had to do
with or was in spite of the long hours.

------
d0m
" I plan to write more about path-finding in StarCraft because there are lots
interesting technical and design bits."

Please, please do so. I find these articles so interesting.. I'm a
Warcraft/Starcraft/Starcraft2 fan and love to see the engineering part of
them. Thanks for writing these articles.

------
j-kidd
The perpetual 2 months deadline is interesting. Is it because 1 month is
definitely too short, while 3 months will invoke the wrath of Boss? I think a
2 months period hits the sweet spot where developers feel like they can get a
lot done, without noticeably impacting the schedule.

------
TimJRobinson
It's been a while since I coded in C and I'm confused as to how you made a
double linked list that can read in O(1) time. Wikipedia has a page in it but
even that says it takes O(n) time, does anyone have an example or tutorial on
how this is implemented?

~~~
zenlikethat
In the article he says:

"All of these lists were doubly-linked to make it possible to add and remove
elements from the list in constant time — O(1) — without the necessity to
traverse the list looking for the element to remove — O(N)."

It seems the implication is that one is parsing over the list already. When
parsing over the list, if you do a comparison that indicates the node should
be removed (i.e. if the unit's health is below 0), you do not have to traverse
back through the list again from the beginning to get the relevant pointer
from the node before the current element in the list, since you already have
the pointer to the previous node in the current node. Likewise for insertion.

But I don't think, strictly speaking, it's possible to have a linked list that
actually has a lookup time of O(n) .

------
pixie_
Great read. I've been there. Write an awesome code base to have someone come
along to replace it all with crap.

------
danbmil99
TL; DR: C++ really sucked back then

------
Kelliot
'so Blizzard wouldn’t have a long gap between game launches.'

You've already lost me

------
Evbn
Why is isometric pathfinding different from square tile pathfinding?
Isometrics is a UI issue, it is still a square grid. Because each tile had non
constant terrain? But that just seems unrelated to isometricity.

~~~
jongraehl
The article describes a scenario where the map editor deals with isometric
tiles, but something compiles them into square tiles.

I think they would have wanted a pathing graph finer than the full isometric
tile anyway. I don't understand how much additional, wasteful pathing
computation resulted from the engine actually using square tiles either.

------
Daniel_Newby
What am I missing about path finding? A simple filter on the nearest neighbors
can weed out paths that are too narrow, then Dijkstra's algorithm can find the
path. After you find a path the first time, it can seed subsequent rounds for
efficiency. So what am I not seeing?

~~~
NickPollard
The biggest problem in games is the number of units - you might send 12 units
through a narrow pass, and the game needs to have the units go through
smoothly without blocking each other too much. From what I remember, the
shipped version of Starcraft still had problems where you'd send a group
somewhere, they'd get a little bit stuck going through, and one or two units
at the back would then re-path to a really convoluted round-trip rather than
just waiting half a second then continuing their original path.

~~~
kzrdude
The bigger units like dragoons and goliaths were notorious for stupid behavior
in groups.. so it was never really resolved, and became a feature that
progamers had to deal with.

