
An Analysis of 155 Postmortems from Game Development [pdf] - akkartik
http://research.microsoft.com/pubs/262301/washburn-icse-2016.pdf
======
13hours
"To avoid schedule slippage, game developers need to spend more time to plan
out all the work that needs to be done so that no tasks are overlooked when
giving estimates. By planning out tasks, you can also estimate each task
individually, and give a more accurate prediction, instead of an optimistic
one."

When things don't work out according to the schedule, there are generally two
reactions :

1\. We didn't estimate well enough, we need to estimate better next time.

2\. Maybe this estimation thing is impossible to get right, let's not put too
much trust in being able to estimate accurately, but rather find ways of
working that don't require estimating all work up front.

I've seen that 1 gradually becomes 2 with experience (the experience of
getting the estimates wrong every time).

~~~
optforfon
I think you're talking about "first year of work" levels of experience.
Estimation is hard - but a major goal of Agile development is to make it
possible.

I thought the same thing as you, but a month ago someone explained it to me in
a way that makes a lot of sense
[https://news.ycombinator.com/item?id=11641316](https://news.ycombinator.com/item?id=11641316)

Actually a lot of the comments about this PDF can be summed up with "practice
Agile development"

~~~
LionessLover
How does that possibly solve the problem being discussed here? Are you the god
who can define ALL tasks (in the agile meaning, a sprint consisting of a
number of tasks) that a large project consists of up front, all the way to the
end of the project 3 years later (for example)? Agile delivers because it
doesn't attempt the impossible.

~~~
elcapitan
The problem seems to be that although companies want to pretend that they are
all agile, the business and planning side really is not, because resources are
not unlimited. So if you can't (even if admitting that it's faulty) put a
number on time and effort, someone else higher up the ladder will do, and then
it will hurt you.

I think that toolchains should have better support for estimated vs used time
and then learning from that. You could classify tasks by field (DB, frontend,
etc) and then calculate correction values based on past failures. Learning
from wrong estimates seems to be even harder than estimating, because it hurts
to admit that you were wrong, so some automated, blame-less version of it (per
sprint or so) might be a good start.

~~~
CuriousSkeptic
The tricky part is that agile is supposed to help those people sleep better.
But it all kind of depends on your ability to explain to them how agile does
that.

Of course budgets and deadlines has to be set. Agile is not a way to ban
those, it's a way to allow them, even if forecasting is impossible. It's about
saying, well at that date I can promise to have kept under budget, and will
have something that more or less reaches the goal of the budget. It will not,
however, be what anyone is imagining right now, let's have fun discovering
what it'll be.

------
eudox
Funny sidenote: the paper mentions a game titled _Age of Ornithology_ under
5.1.g (Scope). Naturally I got excited at the idea of a game where you play
some kind of bird collector, or an intrepid late 19th-century magnifying-
glass-and-butterfly-net explorer and cataloguer of birds.

But unfortunately this title is some kind of cruel joke:

[http://www.phobe.com/sfi/ornithology.html](http://www.phobe.com/sfi/ornithology.html)

[http://www.gamasutra.com/view/feature/130925/schandenfreudia...](http://www.gamasutra.com/view/feature/130925/schandenfreudian_slips_age_of_.php)

~~~
stepvhen
I have a feeling this, and the rest of the games made by "Schadenfreude
Interactive" aren't real, as much as I would love to play Grand Theft Ottoman
or Accordion Hero, or Nazgul Thunder. Hannibal Crossing looks particularly
enjoyable.

~~~
Exras
I saw the word 'Schadenfreude' and was reminded of a satire talkshow on a
radio station in the PC game Grand Theft Auto IV

Found it: Pacemaker - putting the heart back into the healthcare industry.
[https://youtu.be/jG9BgjnpGGY?t=940](https://youtu.be/jG9BgjnpGGY?t=940)

------
norswap
The conclusion:

> Finally, based on our analysis of the data we collected, we make a few
> recommendations to game developers. First, be sure to practice good risk
> management techniques. This will help avoid some of the adverse e ects of
> obstacles that you may encounter during development. Second, prescribe to an
> iterative development process, and utilize prototypes as a method of proving
> features and concepts before commit- ting them to your design. Third, don't
> be overly ambitious in your design. Be reasonable, and take into account
> your schedule and budget before adding something to your de- sign. Building
> o of that, don't be overly optimistic with your scheduling. If you make an
> estimate that initially feels optimistic to you, don't give that estimate to
> your stake- holders. Revisit and reassess your design to form a better
> estimation.

It's amazing how common-sensical this all his. But then again, we could all
use a little more common sense sometimes.

As Charlie Munger would put it: "It is remarkable how much long-term advantage
people like [Warren Buffett and myself] have gotten by trying to be
consistently not stupid, instead of trying to be very intelligent."

~~~
LionessLover
When the results look bad it doesn't mean people don't follow common sense.
For example, when the project would never get approval and a budget with
realism but will get both when risks are left out when asking for money it's
common sense to ignore risks. The team has a budget for another year, and what
happens then is a different problem. The main objective of employees is not to
get projects done, it is to get paid.

Unfortunately our system has created perverse incentives: What manager or
entrepreneur can afford to take a step back and say "what I'm proposing may
not work, let's stop the project"? We have chosen to work under a pressure as
if our lives are at stake (and they are - not our bodies, but our livelihood).
The guy selling worthless shit, getting people to buy stuff they don't need,
that may even be bad for them (or for those producing them) does not have
problems. The same guy voluntarily giving up his job where he would have to do
things he thinks are actually bad and/or useless is seen as a "parasite" on
society - he's not working and we still have to feed him???

You bet all my estimates show the project is worthwhile if I don't have an
(equally good) alternative.

------
Animats
Reading this, it's clear that game development is now a mature technology. I
was working on physics engines in the late 1990s, when nobody really knew how
to do that very well. There were Gamasutra postmortems where the game physics
went badly wrong and things flew apart.

In the early years of the PS3, game projects were failing because the Cell
architecture (a small PowerPC CPU plus six auxiliary Cell processors with tiny
memory) required completely new game engine architectures. That was finally
figured out, but development on the PS3 seemed to run about two years late,
which set Sony back.

Now there are mature game engines, mature physics engines, the graphics
pipeline is in good shape, and the hardware is powerful enough. Problems in
game development now are more about the game than the underlying technology.

(Pre-1990s, game development was a cram job, trying to get stuff done on
limited hardware.)

~~~
greggman
apparently they aren't though. People porting from PC to other platforms are
still finding huge perf issues and have to refactor tons of code. I thought
this gen (PS4/XboxOne) were going to be easy ports but from every team I've
talked to they aren't

~~~
mjevans
A lot of the criticisms I've seen for failed ports describe the exact type of
behavior you would expect from an emulated system.

The ports you reference behave like game systems that used to occupy 100% or
almost 100% of the resource capacity of a dedicated system failing to execute
correctly when they're running in a non-dedicated environment.

They also had the advantage of targeting an exact set of hardware
specifications in other ways and didn't have to scale down, up, or sideways
(other ways of dividing labor) to the way that other platforms work.

In short, they weren't a game designed to scale to a theoretical platform that
then ran on the platforms, they were fit precisely to a single platform and
then weren't designed to be portable at all.

~~~
Narishma
I think you got that backwards. Parent was talking about porting from PC to
consoles.

------
shmerl
_> When looking at games developed for one platform versus games developed for
multiple platforms, we found a few advantages and disadvantages to choosing
one over the other. First, 46% of developers that used multiple platforms
listed art as something that went right, while only 33% of developers that
used a single platform listed this. Additionally, 15% of the developers that
used a single platform listed this as something that went wrong, while only
17% of developers that used multiple platforms listed art as having gone
wrong. Therefore, the evidence shows us that games developed for multiple
platforms may be more likely to have better game art._

I didn't quite understand the correlation between development for multiple
platforms and quality of art. May be those who want to release for multiple
platforms in general care about quality more?

~~~
chrismcb
I don't think this as anything to do with the quality of art. Dealing with
assets is a pain, dealing with assets on multiple platforms is worse. I'm
guessing those dealing with multiple platforms paid more attention to the
process. Where those working with a single platform probably didn't think
about the issues with at a much. Getting the art right, it's probably more
important on multi platform.

~~~
shmerl
Do you mean stuff like making shaders for multiple incompatible platforms or
all art in general?

------
greggman
> Example of poor project management: Leonard Paul of Moderngroove stated that
> Modern Groove – The Ministry of Sound Edition suffered from poor project
> management because they had no dedicated project manager. Most of those
> responsibilities were given to the lead programmer, which resulted in an
> over burdened team. Paul said “I believe hiring a good manager early in the
> project and having clearer deadlines would have definitely helped the
> project.”

> Takeaway: In order to avoid conflicts during the development process, teams
> need to have proper management.

This seems like the exact opposite of Naughty Dog's advice

[http://www.latimes.com/business/technology/la-fi-tn-
naughty-...](http://www.latimes.com/business/technology/la-fi-tn-naughty-dog-
uncharted-20160523-snap-htmlstory.html)

------
fathom
>utilize prototypes as a method of proving features and concepts before
committing them to your design. Third, don’t be overly ambitious in your
design. Be reasonable, and take into account your schedule and budget before
adding something to your design. Building off of that, don’t be overly
optimistic with your scheduling. If you make an estimate that initially feels
optimistic to you, don’t give that estimate to your stakeholders. Revisit and
reassess your design to form a better estimation.

------
xemdetia
This is a strange paper to read because I think I have read most of the source
material, something I certainly haven't felt before in a formal research
paper! I was hoping that it would be more from the context of the publisher
working with Microsoft PC/XBOX developers for new information though. Oh well,
maybe next time.

------
ComodoHacker
I wonder how long before this kind of analysis can be performed by AI just as
efficiently and will be offered as a web service.

------
sillysaurus3
This is excellent for the history and for the examples of what worked and what
didn't. But the interpretations are a little dubious:

 _Takeaway: Game developers should create a well-defined concept before
beginning development as opposed to an ad hoc method of game design._

This would exclude Dota, for example. Gamedev is a world of exceptions.

In general, it's best to playtest every idea you come up with and cut whatever
isn't fun.

~~~
ekianjo
This advice is super generic.

~~~
stepvhen
Sometimes you start game development with a single mechanic and see where that
takes you. Like "what if you could control gravity" and that might lead to
VVVVVV or others. Or it might crash and burn. The advice is to encourage
developers to flesh out ideas and vet them rigorously before starting
development.

~~~
flashman
Sometimes it's complete serendipity. The Rocket League creators were
originally making a deathmatch-with-cars mod for Unreal Tournament. Then
someone dropped a giant ball into the map, and they realised 'car soccer' was
a much stronger concept.

------
DrNuke
Uh? It's not software eng or art but too many lions competing for too little a
pie. CRUDs' marginal return is much better again (or as always).

