
Maxis Insider: SimCity Servers Not Necessary - amelim
http://www.rockpapershotgun.com/2013/03/12/simcity-server-not-necessary
======
chrisacky
"With the way that the game works, we offload a significant amount of the
calculations to our servers so that the computations are off the local PCs and
are moved into the cloud. It wouldn't be possible to make the game offline
without a significant amount of engineering work by our team."

I'm going to point a finger and say that this is clearly untrue, and very easy
to disprove just from basic network monitoring over 5 minutes of playing the
game. I would immediately fire a systems architect that designed a single
player game to compute significant calculations on our expensive [buzz word]
cloud servers. Unless we use different definitions of the word
_significant_...

It's a little ridiculous to suggest that?... plus, for zero benefit (above
DRM), it would have a non-negligible affect on their bottom line if they are
computing time cycles for SimCity on their own servers...

Anonymous source or not, conjecturally, it's hard to not agree with what the
insider has said.

~~~
nwh
It's clear that the quoted text was a lie from the start.

There's nothing you could do in an EC2 instance that I couldn't do on my quad
core i7 at many times the speed and a fraction of the cost. Even if you
matched users one-for-one with large EC2 instances, you'd be looking at
hundreds of thousands of dollars an hour (and that many don't exist).

~~~
firebones
While I agree that this is likely not truthful, I do wonder if part of the
strategy of offloading more work to the cloud be to make an easier transition
to allowing mobile clients (iOS/Android) where local processing capacity is
more of an issue, or to platform-neutral approaches, or to a "take your city
anywhere" play model? (I doubt this is the case because unless they have a
motivation to keep it secret, it would make for a much better explanation for
why they'd want to keep state in the cloud, given the trends towards more
mobile gaming.)

~~~
seanp2k2
EA's claim struck me as really odd too; this just isn't the way that games are
made [yet?] and if they did manage to get something like they claimed running,
there would be much more interesting technical aspects of it that they
probably would have done press releases for. In this day and age, standing up
a system like that is still a major accomplishment, and the details of how
they got around things like processing power and network bandwidth on consumer
connections would be really interesting to the tech community.

TL;DR easily-verifiable claim by EA that stunk from the beginning proven wrong
by anyone who knows how to use Wireshark

Que even worse consumer backlash at EA.

~~~
ZachPruckowski
Forget Wireshark, the article implies that some of the critics tried the good
old fashioned "yank the Internet cord and see how long it takes to break"
method and got 20 minutes of playability.

------
MBCook
The thing I don't understand about all this is why they were unable to new
servers online. After everything hit the fan, they announced they were working
on it and had gotten two new servers up the day before.

Two servers?

It just seems unbelievable to me that the backend was designed in a way that
it couldn't be scaled out any faster than that. Since each region is a
discrete unit, you'd think they should be able to move them between servers.

Was it all intertwined? Did the regions, stats, achievements, and DRM all run
out of the same database? Were they not separate services?

They had to know this game would be popular, they've been pushing it for
months (to great effect). It's a major property and the first release in about
a decade.

Then there is EA. Even if Maxis couldn't figure this out (and I doubt that),
EA has online experience. They're the publisher for Mass Effect, Madden, Fifa,
NCAA, and more. They should have the resources, the people, and the experience
to have prevented this.

If you completely ignore the DRM or the seemingly unimportant always-online
requirement, it this whole thing still seems botched. There were multiple
groups who should have known better and prevented this. My understanding is
that they got some warning signs during the beta.

I would kill for a postmortem blog or article on Gamasutra explaining why they
couldn't scale out faster; to know what decision was the lynchpin that held
them back.

~~~
Afforess
Each game "Server" for SimCity is actually an Amazon EC2 cluster of servers,
with 1 central master DB server. Even when the servers were "full" on game
launch, all of the EC2 servers were responding to requests normally - it was
the cluster's master DB server that was slow. All of the "servers" are
actually in the UK Amazon EC2.

This brings us to the scalability problems and why regions/cities are not
shared to all servers. The database is the bottleneck, so sharing regions
between servers would only worsen performance.

~~~
jaggederest
That makes it even more baffling why they couldn't bring up more servers.

If it's all just a chef/puppet based infrastructure in EC2, you should be
maybe 20-30 minutes away from pumping out a new 'server'. One is as easy as
ten, at that point.

~~~
mbell
We're talking about EA here. You need to include the latency required to go
through enough bureaucracy layers to approve the expenditure of funds on
another cluster.

~~~
chaz
Not to say that there isn't bureaucracy in a company their size, but the
launch of a game this size is a really big deal. There's a tremendous of PR,
and they're getting scathed. Polygon downgraded their review of 9.5 down to
4.0 because of the server issues. I think if they could cut a decent size
check to fix the issues, they would. The problem is likely in engineering,
like a database that isn't scaling.

~~~
mbell
> The problem is likely in engineering, like a database that isn't scaling.

If it were that simple, just cut the number of users per cluster and throw 10
more up.

> I think if they could cut a decent size check to fix the issues

Doubtful within the context of a quick fix, but it is likely the root issue.
See above simple solution that takes 30 minutes to roll out. EA is not a
company run by engineers, its not a company run by people that understand
anything about engineering. What sounds like a simple solution to us that can
easily be implemented by throwing money at it and reaping the customer
goodwill is completely foreign to a company like that. You may as well be
speaking Klingon when you make the recommendation to just throw new clusters
at it.

~~~
chaz
I've met enough people who worked at EA to know they have competent engineers
and managers -- they are not completely inept. When hundreds of millions of
dollars are suddenly at risk, you have a clear channel straight to the CEO to
get the resources that you need.

I'm giving EA the benefit of the doubt that they ruled out 30-minute fixes. I
can't see how any of us can really speculate as to how long it should take to
fix when we don't really know any details. For example, if it was a database
bottleneck, would you commit to walking in and fixing it in 30 minutes? Or
even 30 hours? I think you'd want to know the details, because the scope can
easily be off by 1-2 orders of magnitude.

~~~
wereHamster
> I've met enough people who worked at EA to know they have competent
> engineers and managers -- they are not completely inept.

Nope, nope, nope. Nobody in their right mind will believe you that. If they
fabricate such a game launch then they are complete morons. You simply can not
claim that they are competent after that clusterf __k.

Under "competent" I understand "Having sufficient skill, knowledge, ability,
or qualifications." (definition from wiktionary). Their engineers clearly have
neither the skill nor ability to fix the problems in timely manner. Nor did
(or do) they have enough knowledge how the servers behave under such high
load, otherwise they'd fix it before the launch. And if the problems were
known, then the managers failed to delay the launch. A company which knowingly
releases such a game is not run by competent people.

~~~
cdmoyer
I believe it. All it takes is one higher up executive to pick a launch date
and mandate that it happens, damn the torpedoes. This says nothing about the
competence of the engineers and their managers in the trenches.

I mean, you could question their willingness to work at the company. But,
working on the latest sim city sounds like a fun project and people like
paychecks.

~~~
bsenftner
Bingo! As a former EA senior engineer, my money is on someone in EA marketing
setting the dates and forcing an 80-hour week crunch time for the duration of
this title's production. They probably even brought in "experts" who ignored
foundation code and slammed in their own logic "'cause it works" and now they
have a clusterfuck of logic no one understands. Been there, been screwed by
their managers, and left for good. EA can rot in hell.

------
ghurlman
A friend of mine put it this way: let people run disconnected, offline, and in
the UI show a big glass dome over their city cutting them off Simpsons Movie
style[1]. Over time, gas, food, and water run out, necessitating a
reconnection.

[1] <https://www.youtube.com/watch?v=1kPUGLt1DWQ>

------
drivingmenuts
In respect to the graphics, it looks awesome and I was expecting to be just as
great as previous iterations. I actually wanted to play the game until I found
out it was social multiplayer with cloud-based saves.

That just killed it. I have since bought a copy of SimCity 4 and plan to be
quite happy with it for another decade or so.

~~~
andypants
> That just killed it.

Really? Not the terrible launch?

This says nothing about the game, only your preference of playing single
player games over multiplayer games. There are plenty of great games out there
with 'social multiplayer' or 'cloud-based saves'. These are not problems that
drive away players.

~~~
yebyen
Ostensibly they are, if they are the root of EA's issues. I don't complain
about the servers being down because I don't play games that require servers'
constant attention like that. If they were any other than 'social multiplayer'
then they would be a real-time strategy game, and oddly while those games tend
to sink or swim on the balance, playability, and fun of their multiplayer
side, I too generally prefer to play single player on those games as well as I
would tend to get my ass handed to me by Koreans.

I would rather play Starcraft today than buy SC2. OK so that is stretching,
but I haven't tried the new StarCraft, and it's mostly because of this DRM
situation.

~~~
yebyen
OK. Today I went out and tried Jurassic Park Builder which is basically Sim
City Lite for Android.

I am frustrated because I can't start a new game without destroying my current
progress. I took a slow route, spent all of my coins trying to evolve my
Triceratops too soon, and now I can't use it to progress faster. But this is
only the first day.

It's not as bad as Vector, that makes it clear I won't be able to complete the
game without a purchase. They could be lying. It might be worth the bling just
to own the special tricks. Meanwhile I have come up with some great ideas for
how to use the bitcoin in new and exciting ways. I'm drunk!

------
wmf
I'm not sure how fun "a limited single-player game without all the nifty
region stuff" would be, considering how small each city is. A lot of the fun
is playing at the region level.

~~~
throwaway420
Don't worry.

I'm sure Sim City "Megatropolis" DLC is soon coming that doubles the default
city size. Only $19.95 or whatever they think people will pay.

And don't worry about the traffic jams from bigger cities. Sim City Subway
edition will comes out soon after. Only $15.95.

(yes I am very cynical and angry about this and making bitter sarcastic
comments. This is a game I've wanted for a long time but is so flawed because
of bad decisions)

~~~
meric
I'll wait till I get the "Gold Edition" or whatever that includes all these
expansion packs and the original game for the same price as the game costs
now, a year down the road.

~~~
pseudonym
I think you might be waiting in vain. Unlike most other franchises, The Sims
in particular seem to have an extraordinarily long half-life when it comes to
DLC cost. I'd be rather surprised if Sim City was any different, given how
well it's sold despite terrible PR work.

------
cookiecaper
Patiently waiting for the re-implemented game server and attendant crack that
will let us point SimCity to our own networks.

~~~
seanp2k2
Yep. I assume that the game has fallbacks for "failed save" and "bad data from
server" that you could exploit / patch / use in a work-around here. Worst-
case, you'd make a real server that accepts save game data and stores it, and
maybe does some "verification" of client data packages (AKA "discard client
response and return a big thumbs up in all cases"). It might be more
complicated than that if they're implementing a challenge / response, but I
have no idea how much effort they're going through for DRM these days when
it's been shown to NEVER work.

~~~
robinh
I've done some reversing, and this is EA, so... let's say it's not going to be
something that'll be cracked in a weekend.

------
mattquiros
I could be wrong here, but if Bradshaw's claims were true--that a significant
amount of calculation is offloaded on their servers--wouldn't the game be
unplayable for people who have shitty internet connections anyway? It's easy
for me to imagine SimCity making so many computations to simulate the game,
but for those data to be sent back and forth from the gamer's PC to their
servers? It has got to strain the player somehow.

~~~
ajanuary
Your local machine could be doing coarse grain calculations while their server
did fine grain calculations, and periodically sync up, in the same way FPS's
do (which is why you get laggy players jumping about).

But from what others have said, that doesn't seem to be the case.

------
jasonlingx
Wow people must want to play this game so bad seeing the amount of aggro it's
generated. Makes me want to go out and buy a copy!

~~~
thefreeman
I'm pretty much waiting for the "offline" release, the 100% all clear that
noone is having issues, or the crack. whichever comes first.

~~~
imissmyjuno
I've been waiting for years and i'll wait until the day there is an offline
mode or a patch. It's the principle.

------
jiggy2011
So, if it's not much engineering effort for EA to make the game single player
it shouldn't be too much engineering effort for people who are not EA..

~~~
manmal
I'm pretty sure they reused lots of Sims 1-3 code, and they really have this
simulation thingy figured out. Most probably some really good and generic
engine, or else they could not pump out Sims extensions every other month. I
have quite some respect for this, it certainly took years.

~~~
zevyoura
> I'm pretty sure they reused lots of Sims 1-3 code

You made this assertion elsewhere in this thread also. I don't see any reason
to believe this is the case, do you have any evidence in favor of this theory?
There's a huge amount of PR from them saying this is not the case, and from my
minimal experience with game development it makes no sense.

~~~
manmal
The same amount of PR of them saying that you need to be online because
servers manage these "massive" simulations. No, I have no evidence, would be
cool if someone from Maxis explained the differences between Sims and
Glassbox. What they say here [1] also applies to Sims, only on a bigger scale.
I have yet to see a feature that does not somehow exist in Sims.

[1]: [http://www.polygon.com/gaming/2012/7/4/3135209/simcity-
reboo...](http://www.polygon.com/gaming/2012/7/4/3135209/simcity-reboot-
glassbox-maxis)

------
manmal
Given that Sims 3 runs on my old Windows machine (on low graphics settings),
it's not that surprising that servers are not actually needed to run SimCity.
I don't know if they really use the same simulation engine, but I'd say the
scale is on the same order of magnitude.

~~~
MBCook
They don't. SimCity runs on a new engine called Glassbox. They've been
releasing videos and articles about it for months to get people pumped up.

~~~
manmal
That's what they say, but I don't buy the marketing speak, especially now that
they have been proven liars. Would you throw away all this code and start from
scratch? What they describe about Glassbox was already used in Sims - agents,
resources,... except multiplayer which seems to be just an occasional sync,
right?

------
jellyksong
Anyone notice the interesting old-school-text-adventure error page? It wasn't
404 but some other 4xx page (forgot which) -- it seems to be resolved and gone
now though.

------
hcarvalhoalves
People are complaining as if DRM was the reason they made it like this, EA is
evil, yada-yada. It isn't. It's pretty clear they had a vision for this game
to be a MMO/social game from day one.

The problem stems more from their failure at scaling than not supporting an
offline mode. Diablo was the same crap on launch week.

There's an apparent pressure from publishers to create always-online games,
but something tells me the teams are not having the experience/schedule/man-
power to create scalable architectures to back it. Massive multiplayer gaming
is certainly a hard problem.

~~~
tolmasky
If their vision was for this game to be an MMO/social game "from day one",
then they should have done a way better job sharing that vision and framing
the discussion that way.

The social/multiplayer aspects of Diablo, WoW, and SC are obvious to me, which
is why Blizzard's move never really concerned me _that_ much. Add on top of
that the truly well done matchmaking system and I hardly think about it all.
Doing things _well_ goes a long way in convincing naysayers.

However, when I think SimCity, all I can think of is single player. And the
vague things I've heard about interacting with others seem really lame. Again,
maybe there are some awesome multiplayer aspects, but I have not heard of them
and it sounds like most people aren't that interested in them.

~~~
cbs
_they should have done a way better job sharing that vision and framing the
discussion that way._

In a market where some game's design decisions for online focus and/or online
passes _are_ driven by an effort to reduce piracy and the secondary sales
market, it becomes very hard to convince people that any one game isn't.

Good luck selling that: "No, not in this case, our design just happened to
support them by coincidence, and, oh, we're also part of EA and even though
the cynics have been right about our motives in the past, we're totally not
doing it this time."

------
drivebyacct2
Duh?

The top comment here is more elegant, but EA is clearly lying. Their "apology"
was a joke and amounted to "Sorry we've had so much success, it's your fault
for assaulting our servers."

------
scottchin
Although I'm not thrilled with the launch experience and DRM of SimCity, I
don't think that this article adds any validity due to everything coming from
an anonymous source.

~~~
NoPiece
He isn't anonymous to RPS - they say they know who he is, and verified he
worked on the project just aren't publishing his name. That's a lot different
than an "anonymous tip."

"Our source, who we have verified worked directly on the project but obviously
wishes to remain anonymous, has first-hand knowledge of how the game works."

Either RPS is being duped, or it strongly adds validity to the story. RPS has
a great reputation, and if they say they verified it, I think that puts the
burden of proof now on EA to show that that the claim is incorrect.

~~~
mkr-hn
It's possible RPS isn't known by many on HN. Gaming journalism doesn't often
make its way to this place.

------
larrydavid
As usual with 'anonymous sources/insiders' it's all complete conjecture. The
only people that truly know are EA/Maxis.

EDIT: Seems people disagree that this isn't complete guesswork. Yes the game
runs for a while without a connection, but that doesn't prove that servers
aren't necessary as the headline claims. There is no hard, undeniable
evidence. It's the same story that has been rehashed since the game was
released.

And since the Xbox 720 'leak' fiasco I take these 'sources' with a pinch of
salt, regardless of how reputable the site is.

[http://www.gamefront.com/tech-sites-fall-for-fake-
xbox-720-l...](http://www.gamefront.com/tech-sites-fall-for-fake-
xbox-720-leak/)

~~~
amelim
Well, it's plausible seeing as how multiple people have reported to be able to
play the game while lacking internet connections for up to 20 minutes.
Additionally, the methodology of the code seems to fit with several of the
changes made by Maxis to improve performance (disabling cheetah speed for
example).

Regardless, Maxis has stated they will respond shortly to the article.
<https://twitter.com/rockpapershot/status/311618456640450561>

~~~
eridius
The main thing the article proves is that the servers aren't needed for any
_real time_ calculations. What's to say the servers don't provide valuable
data based on previous events in the game, data that's needed for future
simulation?

