
Why Video Games Have Launch Problems - seattlematt
https://playfab.com/blog/why-video-games-have-launch-problems/
======
jlg23
This article completely ignores that single player, mostly-offline games have
exactly the same launch problems.

The reason games have these launch problems is that publishers got away with
it for too long. Having users who already paid for the game do the beta
testing is much, much cheaper and more efficient than in-house testing.

I sincerely hope that steam's move to grant refunds
([http://store.steampowered.com/steam_refunds](http://store.steampowered.com/steam_refunds))
will sort this out. "Oops, 30% of our pre-alpha-published-as-final purchasers
want their money back". Have one or two publishers file for bankruptcy over
that and it might be a very valuable, albeit painful, lesson for the industry.

~~~
scrollaway
It doesn't ignore it, it's just a completely different issue. General
awfulness of a game on a tight budget and deadline has little to do with
backend scalability issues on large budget games...

~~~
cgriswald
The issue Augustine raises is why games have launch problems. He pins the
blame on backend issues and ignores other issues. (I think primarily he was
just using failed launches as a vehicle to talk about gaming backends, but
nonetheless he argued that backends were the primary cause of failed
launches.)

Another issue the article failed to address is that many games fail
catastrophically when servers cannot be contacted even for online features
that are not necessary for core gameplay; or for DRM purposes. This is a flaw
in the front end that contributes to the same problem.

All those "causes" are primarily fueled by the following:

"It's a major threat to their business. So why does it keep happening?"

This here should have clued him into the larger context. It isn't a major
threat to their business. Getting a refund on a videogame is becoming easier,
but has historically been difficult. Try returning an opened copy of a game to
any retailer. Even explaining that it is the game itself that is broken does
not always result in an accepted return.

The consumer, rather than the business, is bearing the cost of a failed
launch.

So Augustine is correct in a technical sense regarding how backend issues
contribute to failed launches. However, it is the market as a whole that is
the primary driver behind failed launches. There is little incentive to fix
the failed launches regardless of the technical means by which they occur.

------
logent
I've seen two main reasons for this in my experience. The games industry is
one where a 'chase the shiny' business mentality requires developers to
frequently change their target platforms, development stacks, and desired core
competencies for each project. Many of us are jacks of all trades, with little
expertise in things like keeping a service standing under incredible load.
I've seen developers go into this problem with a 'how bad can it be'
mentality, and it never ever ends well.

The other side of things is where you actually have a team capable of
delivering a solid, scaling backend service, who simply do not get the chance
to do so because of poor production planning and overzealous development
schedules handed down from on high. This is not an industry where the
engineers get to define the development schedule or feature set. They can
influence both, but on many teams the managers expect that crunch will solve
all of their scheduling mistakes.

I think you guys at PlayFab are in a great space as this is work that is very
difficult to pull off right and many, many developers will (smartly) choose to
pay for a solution instead of rolling their own.

------
jlees
To address the point of "game developers don't know backends, and backend
developers don't know how to build for games"... I want to think aloud about
the question "how do you hire engineers who can work on the backend of a large
video game?".

One issue parts of the games industry struggle with is simply coding
competency. There's a tendency to hire newer grads and folks who will work for
lower salaries, because people always want to make games. (And why not? Making
games is awesome.) This leads to turnover, poorly thought out (or over-
engineered) designs, lack of "common sense" things like load testing before
launch (remember the industry standards you learned at college? if your
college was anything like mine, you didn't), language soup, etc. And in my
experience, the best backend systems are built by those with at least a bit of
that experience already - rather than gameplay programmers trying to teach
themselves what the CAP theorem is. But what senior engineer wants to work
with legacy spaghetti code, or unlaunched promises (and the threat of future
layoffs), when they could work at Facebook or Google or whoever?

At a previous large tech company I worked at, we built a games team internally
to work on large scale platform stuff (pretty similar to what PlayFab is doing
but aligned with said company's products). It was really cool - we got folks
who were solid engineers but also ex-games industry, or avid gamers,
themselves, advertising team openings via the videogames@ internal mailing
list. We tried to combine the culture/fun of the games industry without the
baggage and conditions.

But platform isn't content, and I recently joined a similar kind of team in an
actual games company (which operates much more like a tech company than most,
since our game is operated as a live service rather than a one-off
downloadable release). Being able to work alongside artists, designers,
narrative writers, sound engineers, event producers etc creates a really
creative environment, and though I'm working on MySQL performance tuning and
internal monitoring data pipelines, I get to hang out and talk about the new
champion releases with folks at lunch. A nice balance, but for those who can't
afford their own platform team, or don't have the carrots to lure us in,
PlayFab seem like a neat alternative. :)

------
njitram
The difference with major online services like facebook and twitter is that
games have a -very- big launch, and get a lot of users at the same time all at
once. It's really hard to test for all that traffic in advance, as there will
be so many configurations, (slow and fast) connections, hackers etc coming in
all of a sudden. Its almost impossible to simulate the enormous variation.

Facebook, Twitter etc begin by a stable (code) base and then step by step
scale up their systems. When something does not scale they can roll back or
quickly fix it. They don't have to ramp up from a few hundred players to
hundreds of thousands in a few hours like in the gaming industry.

~~~
Umn55
" It's really hard to test for all that traffic in advance"

There's this thing, they're called dedicated servers. The reality is that game
companies CEO's/execs don't want to pay for quality when they can push it out
now get the money and patch later, aka. They put in the least effort. The
whole thing is intentional at big companies, a combination of intentionality
and incompetence at small ones.

~~~
njitram
I don't know what dedicated servers has to do with it?

I do not disagree on your point that it might be a combination of intention
and incompetence, although I think most of them really do not want to see
downtime and negative press at release date, when they sell the most. So next
to intention and incompetence there is also a strong incentive for them to
prevent this from happening.

But the orignal point I was making is that scaling up all of a sudden for an
enormous amount of people/traffic is intrinsically very hard, so its easy for
them to underestimate the effort needed, and pay deeply for that on launch
date (and we too as gamers).

~~~
Umn55
>I don't know what dedicated servers has to do with it?

First see the science, I can tell you the facts and figures and you won't
reason to the conclusion correctly.

[https://www.youtube.com/watch?v=PYmi0DLzBdQ](https://www.youtube.com/watch?v=PYmi0DLzBdQ)

If you wanted to avoid "back end problems" (aka problems with 'stressing
servers' because of your desire for locked down games where the server code is
never released to the public) then you should allow gamers to run their own
servers using server software like what games did in the 90's - problem
solved. The corporate world basically stole gamers ability to own and run
servers and since most gamers are 100's of miles away from these game
companies they get away with it.

The free market simply cannot work when buyer and seller are 100's of miles
away from each other, the seller has a huge advantage, all the outraged
customers can't patrol outside the company and give the people there an earful
for being corrupt criminal bastards.

Free market is a fantasy held onto by people who don't know much about how the
real world works and gullibly believe the false ideological propaganda in the
universities.

Protectionism for the rich and big business by state intervention, radical
market interference.

[https://www.youtube.com/watch?v=WHj2GaPuEhY#t=349](https://www.youtube.com/watch?v=WHj2GaPuEhY#t=349)

Crisis of democracy

[https://www.youtube.com/watch?v=ZYFxtNgOeiI](https://www.youtube.com/watch?v=ZYFxtNgOeiI)

Manufacturing consent:

[https://www.youtube.com/watch?v=KwU56Rv0OXM](https://www.youtube.com/watch?v=KwU56Rv0OXM)

[https://vimeo.com/39566117](https://vimeo.com/39566117)

------
flohofwoe
I think the article left one important reason out: you always only hear about
game launches that go bad, never about the ones that go smoothly.

Also: a game launch is like an explosion going off, lots of pressure on the
first day. Marketing usually wants to concentrate everything on that one
single launch event. Demand on launch day and the 2..3 days after can easily
be 100x to 1000x higher than in the second week. You now have the decision to
somehow spin up many times the backend infrastructure that you normally need,
with the associated cost (theoretically achievable with a cloud service, but
see below), even if it is technically possible to smoothly scale up against
millions of players wanting to login simultaneously somebody needs to take the
cost hit, this is usually when the finger-pointing between management and tech
begins. Sometimes it's easier to 'wait it out', since the storm will be over
after a few hours.

As for cloud services: Most game-servers have soft-realtime requirements which
many cloud providers can't provide (at least with the cheap cost plans). For a
web server running at a cloud provider it isn't much of a problem if you have
250ms load spikes or delays from time to time, for an action game it is
catastrophic.

And finally: players that can't login will spend the time raging on the
internet. If you can't get your iOS update on day 1: well, shrug, I'll try
again tomorrow :)

------
devit
> This way in the worst case, the player receives the item in their inventory
> and isn’t charged immediately. In practice, this has proven to occur very
> rarely and has been an acceptable tradeoff.

Except that now the player can spend the money twice in the race window...

That kind of sloppiness by game developers might be related to the launch
problems they have.

~~~
dikaiosune
Not having worked in game dev, it seems to me that some of the industry
problems (high pressure launches, high turnover rate among developers, low
ability to address or prevent technical debt) are responsible for at least
some of the sloppiness. I'd think it really hard to make the right
infrastructure decisions (and implementations) when you have a year or two to
build a multi-million user infrastructure that has to adapt flexibly to any
changing requirements from the game producers/directors/publishers and which
will be thrown away a few years after that.

~~~
krapp
Also, it seems as if the incentive isn't there. Game studios know they can put
a broken product on the market and still make money on it.

~~~
hacker_9
Which is why steam refunds make so much sense - as it's all digital, buying
and refunding are the same couple of easy button clicks for the consumer. If
this where to happen on the mobile app market it would be a game changer
(because of the 100x more indie games there).

------
phn
They caught up with the rest of the "internet-enabled" software. If it can be
updated, the maximum it needs to be is acceptable, not (mostly) bug-free as in
the old days. This gives lee-way to juggle quality with release deadlines.

------
Yhippa
As an outsider is it possible to do game development in an Agile manner?

~~~
pandaman
It is. Some studios actually do this (e.g. Blizzard, High Moon, some had done
it before they had been shut down). The problem is that Agile has been
designed for a process at a service firm, one that does work for hire. As far
as I understand, in that business you don't really want to ship. Instead of
shipping you want to keep your customers engaged and paying your invoices as
long as possible, preferably forever. This can also be useful for a firm that
provides continuous service since there is no shipping involved too. And there
is nothing wrong with it. But games have to ship. Continuous development
without a hard shipping date is called "development hell" [1] in game
industry.

From what I've seen Agile does not work very well here, at least on the
engineering side. Design might be doing something similar to Agile on a small
scale.

1\.
[https://en.wikipedia.org/wiki/Development_hell](https://en.wikipedia.org/wiki/Development_hell)

------
Umn55
Pfft... it's all just bullshit, pressure to release games too early from
managers and hit quarterly profit targets.

That and the obsessive desire for control of everything.

~~~
LoSboccacc
Moreover is not these game have the necessity of being always online. Remember
the bullshit about simcity having to offload computation to servers?

Scalability problem are not because the load of gaming, but as you say because
they are obsessive about stealing every piece of info they can to better
funnel users into premium.

Remember when people could host dedicated server? You would never hear quake
having scalability issues or need to be saved from ec2

------
Mimick
The reason IMO is game developers just don't care since it's a huge work to
micro-optimize a game.

Last years when silicon reached it's limits, all kind of developers start to
worry about the performance. The question "how to make a more advanced
game?"'s answer is not git gud on hardware anymore. It's up to the software
today (again).

------
sergiotapia
For PC I get it, uncountable hardware variations. But for console games, it's
inexcusable. You have one and only one variation of the console. Up your game.

~~~
Narishma
You obviously didn't read the article. That's a completely different issue.

~~~
sergiotapia
The article is mainly about server-side issues with the database, but those
still reflect poorly on the client side.

------
hacker_9
Well if someone created an actually good programming language for game
development we wouldn't have this problem...

~~~
Paul_S
Combination of C, C++ and a scripting language of choice are adequate and work
well. It think business and design problems contribute more.

