
Game developer loses multiplayer service code - eropple
https://steamcommunity.com/games/237870/announcements/detail/1594753310261910516
======
amarshall
The developer added more details in the discussion thread:

> We have all of our available code backed up on our SVN server. But the
> programmer that wrote the lobby code back in 2014 wrote some of the code
> directly to the lobby server. So we don't have these, and since this
> programmer isn't here anymore, and probably wouldn't remember what he wrote
> even if he were, the entire thing doesn't work unless we rewrite. (src: [1])

> Unfortunately, it really is due to incompetence on our part. We really
> didn't know that our server programmer was writing some of the code directly
> to the server until we went looking for it after the server deleted our
> code. We thought we'd just pop the backed-up code directly onto the new
> server and that'd be the end of it. (src: [2])

[1]
[https://steamcommunity.com/app/237870/eventcomments/16330403...](https://steamcommunity.com/app/237870/eventcomments/1633040337745152108#c1639789306538262696)

[2]
[https://steamcommunity.com/app/237870/eventcomments/16330403...](https://steamcommunity.com/app/237870/eventcomments/1633040337745152108?ctp=2#c1639789306567954369)

~~~
onion2k
That really sounds like they haven't been backing up their server, and that
they think of source control as the backup. Both of these things are pretty
bad.

~~~
pmikesell
This is common. Most places use their revision control system as backup for
their source code and take backups of their database. This makes sense because
the source code holds no state. Why would you back up the server? What would
you do when you had 100 servers? You’d take backups of the running disk state
of all of those? To what end?

~~~
onion2k
_Why would you back up the server?_

In case of things like this. Also it's usually quicker to recover a server
using an image than it would be to provision from source in the case of, say,
a hardware failure.

 _What would you do when you had 100 servers?_

Something more scalable. There's no reason why you should pick an approach to
solving a problem and stick with it forever. If the situation changes you do
something more appropriate. That doesn't mean the small scale approach is
wrong when the scale is small.

~~~
ianmobbs
I mean, the better way of doing this is just keep your application code
stateless and make frequent backups of your data stores. If the developers
knew they had server code that wasn't checked into SVN, your solution makes
more sense, but they just had no idea. Even in corporate, if I need to set up
a new box for a service, I don't backup an image from another box and deploy
it to the new one - I just pull my Docker image and run that.

~~~
cwyers
Yes, but part of keeping your application code stateless is that you need to
actually do things to keep it that way. If you haven't done those things, it's
easier to set up Tarsnap in a cron job than it is to make sure your deployment
process is stateless.

------
honkycat
Man, that is really unfortunate. My heart goes out to these developers. I've
worked with a few of these "do whatever works" cowfolk before. It is
exhausting. And, as we see here, can ruin a company.

Personally, I'm getting really sick of the "software doesn't matter, do
whatever you want, ship it" attitude in the industry. It creates a lot of
needless toil because people do not want to learn how to do things properly.

Your software is your product. The primary cost of software is NOT the initial
development: It is the ongoing maintenance. When you cut corners, you are
taking out a loan against your future sanity and time. Higher quality code
bases and better practices lead to stability, ease of extension, and more
money.

___

A while ago I was working for a start-up using Google Cloud SQL Server with
Postgres.

I was relying on the built-in database backup functionality, sleeping
peacefully at night knowing I effectively delegated that task to a higher
power.

Then, one day, I deleted the dev database. Whoops! Hmm... I cannot find the
backups...

So I go onto the google cloud slack:

Me: "Hey, where are these backups stored?

Them: "The backups are deleted with the database."

I felt a bit like Wile E Coyote running in mid-air after running straight off
a cliff.

Needless to say, I dropped all current projects and took a week to developing
a backup, restore, and testing solution for our product. WHEW. Terrifying
stuff!

~~~
neffy
Things I always do a few weeks after starting a new job...

Request the backup copy of a semi-randomly chosen file. It's been
enlightening.

~~~
H8crilA
Related: organizations that would rather pay cryptolocker ransoms than restore
from backups.

------
SeanBoocock
Hard to know what is going on here. They claim that "all the code base got
deleted from its server" where the server in this context is a deployment
target, not a source control server. At any studio I've worked, and even on
larger hobby or student projects, fixing this would involve an automated
build/deployment using <insert build pipeline, maybe Jenkins>. At worst you
could go back to source and build binaries for the target manually and setup
whatever deployment configuration necessary to stand up the service again.

The developer says that they were using uLink for multiplayer which was a
Unity plugin/framework for building services natively in Unity. It sounds like
this was a binary only dependency with no source. Perhaps they didn't store
the binaries in source control (or anywhere else) and would have to re-
implement the networking with some other framework. For a small team and title
with a small player base, it would be hard to justify the cost.

Lessons learned to always have full source for any critical dependencies or a
contract with the maintainer for long term support. Also, make sure you can
rebuild your deployment target from scratch and have your continuous
integration process testing that early and often.

~~~
jayd16
Or just check everything in. It's a bit taboo when you have proper dependency
management to also check in the dependency binaries but I've found this is the
only sure fire way to get rock solid turn key builds.

------
dpcan
In 2010 I didn't know about version control. I released an Android game and
sold it. It did really well. I had released many updates, and hardware was
changing fast.

I wanted to update the game with all new high res graphics for new devices,
and support tablets too.

However, I started working on new functionality in the game, something that
kind-of affected the core of every other thing that was happening in the game.

I never quite got that perfect, but I kept updating, making backups, etc.
Still, no version control.

Then it turned out that the latest update I released about 4 weeks earlier had
an issue with a wide range of devices that someone finally brought to my
attention. This was a new Samsung device that everyone was getting.

Customers started writing in about the game not working on their new device.

I had a backup of the code before I released the last update (which had a ton
of changes and new features), and then a backup AFTER the release but also
AFTER I put in a lot of the broken core functionality.

Basically, I was in a situation of having an 8 week set back to redo all the
changes I made before the last release from a backup. That would take about 4
weeks, then I'd have to release, and start again on the changes I've been
spending 4 weeks on.

The game was doing well, but I definitely wasn't 2 months ahead in income, and
this was a side project.

I ultimately couldn't do anything about it and had to just press on with the
new changes that did go out, but in a less than stellar way about 4 weeks
after that.

In the meantime, I had to make a ton of refunds for people with certain
devices, and then block that device from the dev console and put notes in the
description not to buy the game if you are using a device like X.

Doing this basically brought my sales to a halt.

I ended up putting out one more update with the small changes I had that
weren't perfect, people didn't love the change, I did make it so people with
the new devices could download and buy again, but my rank and sales hand
tumbled miserably anyway. The game sold a few copies here and there for a few
more months, and then within a year I stopped development completely and
stopped selling the game.

So..... version control and proper backups are VERY important, especially if
you have customers. This game was my ticket to being an independent game
developer making a living selling games, and then it crashed because I
couldn't work on an older version.

~~~
ggg3
the problem here is not even your process. its google/apple walled gardens
that do not the end user install a freaking older version!

~~~
echelon
I'm not sure why you were flagged. While the OP is responsible for a lack of
version control, the distribution channel is very inflexible and something
developers have to fight with.

Many other distribution channels provide multiple binaries or versions and let
the customer choose which to install.

Apple and Google are very simple and primarily aimed at non-engineer
consumers, so it makes sense that they wouldn't allow people to select a
version. It doesn't mean that this doesn't suck.

~~~
Siwka
Maybe things have changed now, but as far as I can remember Google Play does
let you deploy older APKs. But it's just from the developer side, it's not
like users can choose which APK to download.

------
tyingq
So I guess no version control where at least one developer had a checked out
local copy? Ouch.

~~~
eropple
The comments suggest that while most of their code is in SVN (and to forestall
the usual discussion, SVN is totally defensible for game devs, most people
don't know about Git LFS and it isn't a perfect solution), a tools developer
made edits in production to make the lobby code work and he's no longer with
the company.

~~~
amoshi
The comment in question

> zede05 [developer] 26 Jun @ 3:58am @The sap is rising! We have all of our
> available code backed up on our SVN server. But the programmer that wrote
> the lobby code back in 2014 wrote some of the code directly to the lobby
> server. So we don't have these, and since this programmer isn't here
> anymore, and probably wouldn't remember what he wrote even if he were, the
> entire thing doesn't work unless we rewrite.

~~~
SeanBoocock
Lessons learned but that is pretty unbecoming. This isn't a hobby project or
something that was released for free. It's definitely not representative of
most studios in the video game industry.

------
leowoo91
[https://steamcharts.com/app/237870](https://steamcharts.com/app/237870) some
stats for the game if relevant..

~~~
argd678
37 CCU avg is a dead game, that’s admirable they put a best effort into making
their last players happy. That also means, while a little embarrassing to lose
code, it’s not a big deal. And it’s also not surprising they’d edit the code
on the server and not have it in SVN, game development is full of sins like
that.

Update: corrected to CCU from MAU

~~~
arkem
For context the current CCU makes it the 1,898th most played game on steam
right now (out of 10,772 games that have at least one player).

The drop off comes very quickly!

1st most played game has 694,472 CCU (Counter-Strike: Global Offensive)

25th most played game has 19,744 CCU (Terraria)

50th most played game has 13,172 CCU (Counter-Strike)

100th most played game has 4,367 CCU (Sims 3)

500th most played game has 529 CCU (Octopath Traveller)

1000th most played game has 167 CCU (Shadowrun Hong Kong)

10000th most played game has 1 CCU

~~~
kibibu
OT, but I'd be interested in where Fortnite, Minecraft, and League of Legends
would hypothetically fit in that list.

Heck, I think Roblox would have them all beaten.

~~~
arkem
In 2014 Riot released numbers of 7.5MM peak concurrent players for League of
Legends (off a MAU 67MM) and in 2016 reported a MAU of 100MM (but no new
concurrency figures). Those are old stats but still useful for comparison.

Disclaimer: I work at Riot but don't have any internal numbers to share.

~~~
arkem
Wanted to jump back in here now that new public information is available, for
August 2019 there was a daily average of 8MM peak concurrent players.

[https://na.leagueoflegends.com/en/news/game-
updates/special-...](https://na.leagueoflegends.com/en/news/game-
updates/special-event/join-us-oct-15th-celebrate-10-years-league)

------
Causality1
That's really a shame. I sunk about a hundred hours into that game when it was
new. The building mechanics in it were so robust it effectively contained
SketchUp right in the game for building weapons, devices, vehicles, etc.

[https://i.imgur.com/VqFuxWw.jpg](https://i.imgur.com/VqFuxWw.jpg)

------
wpietri
Unless you regularly test your backups, there's no reason to think you have
backups. Unless you regularly rebuild from scratch, there's no reason to think
you can rebuild from scratch.

~~~
gchamonlive
That really depends.

If you use cloud solutions, like aws, instance images are reliable backup
units without having to test it. Managed resources like aws databases (rds,
dynamo... ) have built in backups that just work.

You only need to proof test your backups if you are using untested or 100%
handcrafted solutions, and in those cases you either know really well that you
are doing or you should be using battle tested solutions

~~~
kemitche
I trust RDS to backup the DB properly.

I don't trust myself to have perfectly configured RDS backups; to not have
missed something critical (S3 bucket); to not have any number of other things
outside the DB itself that may come into play should the worst happen.

Running through a "restore from backup" exercise every so often helps suss out
where the missing pieces are.

------
carapace
FWIW...

"Backups are a tax you pay for the luxury of restore"

"How Google Backs Up the Internet"
[http://www.youtube.com/watch?v=eNliOm9NtCM](http://www.youtube.com/watch?v=eNliOm9NtCM)
Detailed notes: [http://highscalability.com/blog/2014/2/3/how-google-backs-
up...](http://highscalability.com/blog/2014/2/3/how-google-backs-up-the-
internet-along-with-exabytes-of-othe.html)

------
fragsworth
Here's my take - it looked like their average concurrent players was around
~30 before this happened, which indicates it was nearly a dead game. It's
free-to-play, and I can't say how much they made per player, but a decent
amount would be $0.05 per DAU. Converting concurrents to DAU is difficult, but
let's say they had 500-1000 DAU. That would be on the order of $25-50/day.

My guess is they _could_ fix the problems, but it would cost an engineer a
month? or a couple months? Considering the game was on the decline, it's hard
to justify spending any effort to fix the issues.

Now the weird part. It was a F2P game - I don't know how much stuff people
were buying in the game, but there's an implied contract that when you
purchase a digital good, the digital good will continue to work for some
reasonable amount of time. They are breaking that agreement by not restoring
their servers and I am not sure what that means legally - there's potential
for class action lawsuits, maybe?

~~~
jsnell
It was only made free to play after the blog post. Before that it was a $10
game: [https://steamdb.info/app/237870/](https://steamdb.info/app/237870/)

------
eeZah7Ux
To all the commenters that whine about using version control: version control
is not a replacement for backups and disaster recovery procedures.

Repositories get deleted by mistake. Company accounts get closed. VCS servers
break down or catch fire or get wiped maliciously or by mistake. Disk
encryption keys get lost.

Offline backups on remote locations exist for a reason.

------
tempsolution
Hilarious. Hobbyist projects for the win! In general it is helpful not to be
completely ignorant of how the state-of-the art in your line of work is done
and perhaps ask why it is done this way.

Its almost inconceivable to me how it is possible that a company, no matter
how small, does all these things so utterly wrong that shit like this is
possible.

Even as a child, when playing around with Delphi and Turbo Pascal, I made my
backups dutifully onto an external drive. I did not use version control
because I had no clue what that was (and, arguably anything is better than
SVN), but at least I had backups of everything.

------
rs23296008n1
I'd like to know more details about this but I sense we have what's important
already.

Version control and backups are basically a form of insurance. Sometimes even
a complete backup doesn't help. Even losing a week on a fast changing code
base can set you back a month. The path you take forward has value as much as
the destination.

Insurance is completely useless right up until you need to claim. Then it is
the most important thing ever and suddenly it is absolutely critical to
getting back on track.

Lessons and reminders for almost anyone doing almost anything creative.
History forms your memory.

------
caseysoftware
The lesson here isn't "back up everything" but really is (and has always been)
"make sure we can restore the system"

Because backing up is good but totally irrelevant if you don't occasionally
use on of those backups to restore back to a good state.

One quick an easy way to test that is to have new developers apply the
deploy/restore system to their own machine using a recent backup. (Obviously
scrubbed of sensitive info.)

You get a developer online faster, another set of eyes on the process, and
validation that it works. It's not perfect but pretty good.

------
wolco
I don't always use version control.

But I understand when I don't I'm taking the risk that this will disappear
with no recovery. Most of the time that's okay, some scripts have low
reusablility and somethigs I need an excuse to refactor.

To run a business without it is inviting failure. Once you involve someone
else the conditions to recreate get hazy.

It sounds like only one piece wasn't in version control. That might be why
that developer is no longer there.

~~~
pizza234
I religiously use version control, and additionally, I use PRs and issues even
on projects I work on alone.

It's not about their purpose in itself, rather, it's about a structured
process and discipline.

All in all, source control (and even PR/issues) virtually cost zero, so while
there are arguments again unit testing/documentation etc., there's hardly an
argument against it.

Besides, no source control, no bisect :-)

~~~
theon144
Issues I get (and practice), but PRs? Are you requesting a PR from yourself,
approving the PR when you have meditated about it for a day? :)

~~~
codycraven
I could see being able to look at your code diff outside the context of your
editor could help you identify issues you wouldn't otherwise see. Especially
if your brain flips into code review mode when looking at a GitHub PR.

------
SteveNuts
How do they have the resources to write a new game but not rewrite the MP
code?

~~~
eropple
Older game. Smaller playerbase. Probably bad ROI.

~~~
ZoomStop
A lot of people paid $24.95 for this game, and now they are going to abandon
it, announce a sequel, and make it free for anyone who wants it. It's a triple
slap in the face to existing customers. They should have re-wrote the needed
code or contracted it out.

~~~
eropple
One thing that _absolutely_ has to end is that the idea that a game has now
become free is a "slap in the face" to customers. Customers paid for the game
then-and-there, where they bought it and when they bought it. Discounts are
not "insults" to those customers and by extension neither is making the game
free and (possibly?) open-source.

As for rebuilding the multiplayer: it sounds like they had about 30 concurrent
players and minimal ongoing sales. That's a dead game and at that point, "cut
bait" is not unreasonable; they're not exactly a giant developer with money to
splash around. "Give players free keys to the new game and move on" is about
as fair a path forward as they can make, and is doing pretty right by those
players.

I didn't submit this story here to get the cheap-games-earn-a-lifetime-
commitment crowd riled up. (And make no mistake: $25 is a _cheap game_. So's
$60, for that matter.) I posted it because it's a great object lesson to _back
up your stuff_. If I can be honest, I'm mildly regretting doing so after
reading your post.

~~~
philipov
The practice of releasing a game in early-access undermines the notion that
you are paying for a game there-and-now. Even games that get 1.0 releases are
often incomplete or continue to receive free content updates. Planet Explorers
was released in early access, so people who bought it were clearly investing
in its future, not paying for it as it was then.

This kind of ambiguous relationship being pervasive in the industry muddles
traditional notions of when the studio's commitment ends, which opens the door
for misunderstanding and abuse.

~~~
eropple
PE's 1.0 came out in November 2016. It's well and truly out of Early Access
and has been for nearly two years _and_ the playerbase had dwindled to a
rounding error.

And even with Early Access, you're still paying for it as it is at the time of
purchase--you're buying for the now or for the hope of the future, you are not
buying for any future commitments of any sort. If this isn't well-understood
already, it had best become understood, because financial exigencies don't
really leave a lot of room for "but I bought it in Early Access!".

The finances of games are brutal and punishing and the race to the bottom that
consumers have happily encouraged has resulted in those consumers' investment
not being valued nearly as much. This was foreseeable and is inevitable. The
best way to be reasonably assured that the games you like continue to be
worked on, maintained, and managed is 1) pay a _lot_ more for them up front,
or 2) play games with ongoing subscription systems. As-is? It is economically
non-viable to throw money after 30 concurrent users and nobody with a pocket
calculator could fault them for it.

------
ivanhoe
And God knows how many other games and apps out there are happily running
right now not knowing that some guy edited code directly on the server and
left no trace of what he's done...

------
based2
Does a GIT binary

    
    
       without history rewriting exists?
    

Immutable, with only an add history mode

or with a CQRS filtering management.

~~~
AlexCoventry
You can disable garbage collection in git, which will keep all versions of the
history accessible by the reflog, in principle.

[https://stackoverflow.com/questions/28092485/how-to-
prevent-...](https://stackoverflow.com/questions/28092485/how-to-prevent-
garbage-collection-in-git)

------
GreaterFool
Maybe they just really didn't want to support it anymore? Sunset the
service...

