
“What Went Right and What Went Wrong”: Analysis of 155 Postmortems from Gamedev - winterismute
http://research.microsoft.com/apps/pubs/default.aspx?id=262301
======
Negative1
The old post-mortems in Game Developer magazine were written almost in
template form.

Went right: 1) Great team/culture 2) Everyone put forth the effort (scheduled
crunch) 3) Some amazing techwork or artwork or designwork etc...

Went wrong: 1) Didn't anticipate x/y/z 2) Burnout (remember that crunch)? 3)
Bad/lazy scheduling and/or capacity planning

Kudos to the people who did this at MS but you're looking at tainted data.
Most game developers know what really went wrong and do not publicly broadcast
it in fear of losing future contracts or publisher trust. In the end the
postmortem becomes a soundboard for 'shoutouts' and cheer-leading instead of
that raw, unbiased feedback.

I've been to all hands post-mortems that were like that and it can get really
really ugly.

~~~
ap22213
What are the _real_ reasons for successes and failures? You left me hanging...

~~~
jonny_eh
People making mistakes, of one type or another. In these post mortems though,
they rarely attribute mistakes to any individual, or even team.

------
teddyh
> _9\. CONCLUSIONS_

>

> _We find that we were able to identify both best practices and pitfalls in
> game development using the information present in the postmortems. Such
> information on the development of all kinds of software would be highly
> useful too. Therefore we urge the research community to provide a forum
> where postmortems on general software development can be presented, and
> practitioners to report their retrospective thoughts in a postmortem._

>

> _Finally, based on our analysis of the data we collected, we make a few
> recommendations to game developers. First, be sure to practice good risk
> management techniques. This will help avoid some of the adverse effects of
> obstacles that you may encounter during development. Second, prescribe to an
> iterative development process, and utilize prototypes as a method of proving
> features and concepts before committing them to your design. Third, don 't
> be overly ambitious in your design. Be reasonable, and take into account
> your schedule and budget before adding something to your design. Building
> off of that, don't be overly optimistic with your scheduling. If you make an
> estimate that initially feels optimistic to you, don't give that estimate to
> your stakeholders. Revisit and reassess your design to form a better
> estimation._

~~~
padobson
There's some world-shaking insight.

The best piece of advice I can give is to never, ever give an estimate to
stakeholders until you've worked on the estimate first. And don't finish an
estimate until you've actually built some small proofs of concept.

Your estimates will still be wrong, but they'll be much more accurate than
some off-the-cuff number that your stakeholders will be building their various
plans around.

It all sounds straightforward, but the hardest discipline for me as a
developer is to keep from saying things in stakeholder meetings that make my
stakeholders happy in the moment, but ultimately needed more thought and
effort. It's something I struggle with in every such meeting, to varying
degrees of success.

~~~
hluska
I think this is a symptom of a fear of "I don't know." If I could change one
thing in dev cultures, I would instill a healthy respect for the sentence:

"I don't know, but here's $how_I'll_find_out and here's
$when_I'll_have_an_answer."

~~~
ChuckMcM
Exactly right. Too often I've seen people commit to a schedule where the
milestones were of unknown difficulty. When cutting through a new jungle, you
have no idea what is between you and your goal, so really all you can say is
that the best you can hope for is X if this jungle happens to be like previous
ones.

This is especially true in startup situations where you are learning
technologies that may themselves not be complete, with people who are learning
their own new things, to achieve a result which is only hypothetically
possible. It drove my CFO nuts but rather than commit to a schedule for a big
deliverable I would walk backwards from the end point and say, "These are the
stops between where we are, and where we are going. We measure our progress by
getting to each stop, but like a subway map there isn't a known amount of time
between stops, only what the stops are." And he would come back with "well we
only have money to get to this date, will it be done by then?" and then we
would talk about the uncertainty between each of the milestones. At some point
you can reach a common understanding of what the unknowns are and how
discovering their value will inform on the difficulty of the next step.

That said, I've met managers who just say "Oh we will be done by %x date." and
then basically worked the problem the same way I have.

If you're agile you can break down the intermediate stops as sprints but
estimating the backlog is still the killer step.

~~~
brianwawok
It's not agile to list out 32 milestones on the way to completing a project
and estimate them up front. It's agile to list 32 steps and estimate the first
step, which you will still get wrong.

------
vvanders
I found the Games Outcome Project to be much more rigorous(and anonymous) on
the what practices help and hurt gamedev:

[http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Ga...](http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Game_Outcomes_Project_Part_1_The_Best_and_the_Rest.php)

~~~
pandaman
If you compare the conclusions, this topic's research, while obvious, is
closer to reality that the one, which concluded that crunch literally "makes
games worse" despite the fact that every single game, which made a significant
critical/financial impact had a period of crunch.

~~~
hnbro
> despite the fact that every single game, which made a significant
> critical/financial impact had a period of crunch

this argument doesn't necessarily disprove the notion. it's plausible to
suggest that whatever impact they had was mitigated to some degree.

~~~
pandaman
It sure does not. My concern is that in this research they correlate the
amount of crunch with quality of a game and show that "better" games have
statistically less crunch. Since I know the games, considered successful in
the industry, all have a lot of crunch, it sounds like they use their own
criteria for quality, which has nothing to do with the common meaning.

------
ekianjo
The recommendations at the end of the pdf seem to be written by captain
Obvious. They are so generic they can apply to about ANY project even non
gamedev related.

~~~
Bahamut
One of my friends who just left being full time in the game industry pointed
out to me that the game industry is 10 years behind in software development
best practices. Churn and burn (like many startups) is a daily reality.

~~~
jblow
If your friend was not very senior (10+ years experience), I would not believe
what he says, since he probably did not have enough experience to judge the
situation.

I think the idea that the game industry is "behind" other fields is kind of
comical, given that games are some of the most complex software in the world,
and big game teams have only a few hundred people on them, and meanwhile
something relatively trivial like Twitter has 4000 people. It's true that game
teams don't do a lot of Agile or TDD or whatever the next buzzword is, but
that is because those things are mostly superstition and obviously don't work
when you start attacking hard problems.

So if you are someone a few years out of school who learned TDD it is easy to
say "games are behind, they don't do all the new stuff!!" while being unaware
that almost all the new stuff is bogus cargo-cultism anyway.

I do agree that the game industry engages in unhealthy levels of crunch that
are to its long-term detriment, but this is mostly an orthogonal issue to
software engineering practices.

~~~
hythloday
I was in the games industry from 1999 to 2012 (and then left to go to
Twitter!) and I think his friend is absolutely right, the games industry
shoots itself in the foot repeatedly by believing that it's a special
snowflake full of "hard problems" that no-one else tackles and continually
(and badly) tries to reinvent the wheel. The universal culture of deathmarches
and the denigration of mainstream software engineering and project management
techniques are not in any way coincidental.

(I loved the Witness, btw.)

~~~
pandaman
If only game studios could raise money on negative profit margins instead of
being forced to deliver milestones to be able to run regular operations...
Every day I'd be writing unit tests and doing standups in between code reviews
and updating my GitHub profile.

~~~
hythloday
If you're comparing them to VC-funded startups, games studios run on negative
profit margins until the game is released!

~~~
pandaman
Some might, but it's not a common business model. Most game studios work for
publishers and they sell the game continuously in form of milestones. Even a
studio owned by publisher has to continuously deliver or it will be shut down.
Twitter, which I am comparing to, just gets money on the promise of the future
profits. Correct me if I am wrong, but I don't believe it has any software
deliverables it has to meet every quarter in order to make payroll.

~~~
hythloday
Twitter, now, sells advertising, which is 90% of its revenue. While it was VC-
funded, which is really what you should be comparing it to as it's the
prevalent stage of an startup, it raised money from VC funds by hitting non-
contractual milestones in terms of MAU/DAU increases. I don't think that's
particularly different to the milestone model that's prevalent in games - a
third party pours money incrementally into the development of a product
contingent on the mutually agreeable evolution of the product.

Unless you're in the habit of delivering RC-quality milestones (which is afaik
unheard of in AAA development), the marginal value of a milestone is negative
right up until the last one.

~~~
pandaman
I am sorry, it seems that my point is completely lost behind snark. Let me put
it straight then.

The argument in this thread is that games industry is somehow behind times and
not using the best practices used at places such as Twitter. The question is -
how do you know your practices are the best and not the other way around?

Game studios go out of business if they don't deliver quality software on time
and on budget. I know it first hand since I have been through several studio
closures. The practices game studios use are tested through natural selection
- if they fail at delivering software they are out of business. To make things
more interesting there is also competition: a good game decimates sales of the
worse games released around the same time. And if the sales go below the
projected ROI - it's the game over. So it's not enough to be good enough to
survive, you need to be better than the competition.

Trendy web companies don't sell software. They sell services and the quality
of software they use is secondary. E.g. if I wrote a Facebook's clone but 100
times faster, using 10 times less memory and with 1/1000th of Facebook's staff
it would not threaten Facebook. People would not close their accounts and move
to my network just because I have better software. A web company is fine as
long as their website runs semi-reliably.

So how come the battle-tested practices of the games industry are so bad
compared to the practices of the industry, which mostly sells advertisement?
What are the criteria you use to compare?

~~~
emelef
Games are more like web products than you think, I'd say. If you come up with
a game-stack: 3d engine, netcode, the works, that vastly outperforms a game
like League of Legends, that does nothing to threaten them either.

I'm not siding with this notion that Twitter (or webdev in general) is ahead,
mind you, but it's fair to say that neither websites nor games are about
selling software. They're more about entertainment.

~~~
pandaman
You are right, f2p games are a lot like web - they sell service, not software.
Interestingly, these companies look a lot like web firms - thousands of people
working on the product, which barely changes and, I've heard, using the same
programming practices. This is not what people mean when talk about games
industry.

------
jacques_chester
PDF is on the right, for anyone else wondering what to do about an abstract
that doesn't contain any spoilers.

~~~
brudgers
Link: [http://research.microsoft.com/pubs/262301/washburn-
icse-2016...](http://research.microsoft.com/pubs/262301/washburn-
icse-2016.pdf)

------
chipsy
In the past, Gamasutra has also published data indicating correlations between
"biggest problems/biggest successes" and marketplace success.

The takeaways I remember about that: A good producer matters, a team of a
least three people does better than one or two, and experience shipping stuff
matters.

------
msane
Apparently they didn't achieve any insights suitable for mentioning in the
summary.

