
How the EVE Online Servers Deal with a 3,000 Person Battle - JGM564
http://penny-arcade.com/report/editorial-article/planning-for-war-how-the-eve-online-servers-deal-with-a-3000-person-battle
======
rauljara
The report of the battle they link to (<http://www.pcgamer.com/2013/01/28/eve-
online-battle-asakai/>) is even more fascinating to me. You spend hours and
hours of effort in order to get a ship, and one way of thinking about them is
as the time spent to earn them compressed into 3d form. Unlike (most) other
MMO's, battles actually destroy things. So in that youtube clip, you're
watching years and years of effort getting evaporated because of a mis-click.

In most of the games I play, the pay off of the grinding is getting to enjoy
fighting more. But I could never play EVE, because battles would be so anxiety
provoking. There would be no pleasure at all in earning a Titan, because I
would be forever terrified of losing it.

Of course, all games are kind of there to destroy your time. But for some
reason it doesn't feel quite as wasted when you have a virtual thing there
that is bad-ass in proportion to your time investment.

~~~
fatbird
That feeling of "My ship! My beautiful, beautiful ship!" is offset by the fact
that major alliances tend to have strong reimbursement programs. If your
dreadnought goes boom in a fleet op, you get the cost of another hull and
gear, and get to draw on corp/alliance stores as well.

Otherwise, they'd never get their supercap pilots out of the hanger :)

~~~
baddox
What would be the point of having all this cool stuff if no one else even gets
to see it and you don't get to use it?

~~~
fatbird
If you spent six months grinding the ISK to buy and outfit a carrier, how
willing would you be to take it into a battle where any of a hundred things
could destroy it, including server lag or a random disconnection?

~~~
baddox
I would invert the causality. Why would I spend six months outfitting a
carrier if I had no intention of using it in combat or even flying it anywhere
where other people would see it?

~~~
glesica
Same reason people spend hours tending their farmville farms...

~~~
baddox
I've never played Farmville, so I probably shouldn't speculate, but I'm under
the impression that it is at least ostensibly a social game where you invite
people to help with or at least view your farm.

~~~
glesica
Yeah, but so is EVE and other games like it. I have never played EVE for any
significant amount of time, but I did play WoW for awhile and there were tons
of things that people would sink time into for purely social reasons. I
imagine EVE is the same way. You don't just want a Titan because it is useful
in the game, you want a Titan so you can have a Titan.

------
jere
[Edit: this comment turned out not to really be about Eve and more about the
difficulty of real time MMOs]

This blows my mind. I've been thinking about writing a small multiplayer game
for a few weeks. And today I was actually doing back of the napkin
calculations. The thing that quickly became clear is bandwidth is O(n^2) where
n is the number of players in the same location, since you have to share the
location and velocity of each player with each other player.

Here's an example of a calculation I was using. It's the monthly bandwidth in
TB of having a certain amount of players in a shared space (3000 in this
case), assuming that the location/orientation/velocity of each player's ship
is contained in a measly 48 bytes and you attempt 30fps.

(48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB

THIRTY THREE PETA BYTES to simulate a single battle for a month.

I'm looking around at various VPS providers and the bandwidth they offer.
Using the naive calculation above and the cheapest Linode offering and it'd
take $3 million/month to support a single one of these ongoing battles.

Now, I'm sure the numbers above aren't used in reality. Obviously, you can
reduce the updates per second... but not by much. You can't shave much off how
much you send. It at least gives you a sense of the orders of magnitude
required to support such a thing. Even contemplating supporting 100-1000
people at once is looking very difficult for me to pull off.

~~~
gnarbarian
I think one thing Eve has going for it (and this is only an assumption) is
that it wouldn't need to sync each player's position on each frame. It could
reduce the load dramatically by only sending the acceleration / orientation
data for a ship.

So if a ship has set a course and has stopped accelerating it doesn't need to
send anything else unless that course is changed.

~~~
jere
I haven't really played Eve, so I can't say. I'm imagining ships flying around
and changing course constantly in a battle, but if someone can give a better
description, I'd appreciate it. It'd be further reduced by several players
sharing a ship... no idea how common that is.

What I'm describing is definitely leaning more towards players that are
constantly moving around though. It helps explain to me why in Asheron's Call,
for instance, when you had too many people in the same city a "portal storm"
arose and people started getting teleported out of the city randomly.

~~~
afterburner
If you check the video you'll see it's very slow paced and the ships basically
don't seem to move at all (maybe tiny unseen ones do I don't know)

<http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/>

~~~
gknoy
Part of the reason it is so slow is that (as mentioned in the article) the
server slows time down (as far as 10% of normal) in order to ensure that
nothing gets lost.

------
RickHull
Fun facts: This fight was between the Honeybadger Coalition (HBC) and the
Clusterfuck Coalition (CFC). The core of the Honeybadger Coalition is TEST
Alliance, and the core of TEST Alliance is Dreddit, the "original" Reddit-
based corp (there are / have been several, some of which predate Dreddit and
have since folded).

HBC routed the CFC, which has Goonswarm as its core, which is organized around
the Something Awful forums.

TEST Alliance was nurtured by the CFC for a long time, splitting off only in
the past year to forge a more independent identity. As such, the hostilities
between these old friends is hard to gauge. At this point, the presumption is
that it's all in good fun. But that may change.

~~~
alex_c
Admittedly I haven't followed Eve politics for over a year, but I would be
very surprised to see Test and Goons get in a "real" fight (e.g. over
sovereignty or revenue-generating moons). They both treat the game in the same
way (a strange mix of competence, incompetence, and irreverence) and have too
much shared history. I suspect the only reason they're even hostile right now
is because they have absolutely no external threat and are bored.

~~~
Semaphor
Fits with what I read on Reddit when it came up. Was a fun fight and TEST won
but no one is really mad at anyone else:)

------
stephengillie
It's like each solar system is a VM, and they can't move their VMs to a new
physical server without disconnecting all clients. And all of their physical
servers are at 100% load all of the time? Oh, I guess it's 100% utilization,
not 100% load. As in, they don't spin down servers to save power during off-
hours I guess.

The time dilation is a neat solution to the server load problem, but it's sooo
annoying as a player. In beta, it was interesting to watch the entire game
desync and grind to a halt, but we could still chat and look around.
Interesting, but frustrating.

~~~
alexkus
Two main solutions (given that the article mentions that they have huge
machines already):-

A) Allow a single solar system to span multiple machines. Very hard,
especially if the server software isn't architected for this. Retrofitting
this can be nigh on impossible.

B) Have a few _huge_ machines that can be used to host scenarios like this
and, more importantly, have a way of migrating users over to the huge machine
seemlessly.

The latter can be done but it's tricky, especially if transferring game state
between instances of the server is not simple (I'm not talking about
transferring the VM itself with something like vMotion). It comes down to:-

1) Being able to make the bigger machine act as a temporary proxy pushing
connections data back to the smaller machine.

2) Having a way of telling clients to make a new connection to the bigger
machine and, once that connection is made (and the data is being proxied to
the smaller machine) cut the connection to the smaller machine. Users see no
loss of service or reconnects at all.

3) Once all clients are now being proxied by the bigger machine; pause and
transfer the game state from the smaller machine to the big machine and then
continue. Obviously it works best if a chunk of state can be transferred in
the background and then the final transfer (and pause) is as short as possible
in order to transfer over the bang up to the minute state.

Option (A) is always the proverbial "In v2 of the server we'll do it a
completely different way..."

~~~
dclowd9901
Option 4C) Create a raspberry pi cluster farm, and host every player with one
microserver.

~~~
kalleboo
How would the microservers keep up to date with each other fast enough to be
useful? "eventual consistency" doesn't work with real time strategy games.

~~~
dclowd9901
How would they not? Latency between servers would be near zero, and if they
all handled point to point transactions while broadcasting their state on each
transaction, you're getting all the power you need. We're not talking about
complex processing here. It's an RNG coupled with simple health and xyz.

~~~
kalleboo
Broadcasting their state to what? All the other nodes? So each "microserver"
would still need to handle thousands of connections and data for every player?

------
ChuckMcM
This is a nice article.

A friend of mine did some research for DARPA/NSF on internet 'crowds.' His
research was looking at the question of fractal gatherings, which was
basically you were 'near' the people around you in the virtual 3D space and
could hear what they were saying, you were adjacent to people who were one
space away but could hear them if they 'shouted' and the crowd was at a
gathering that was sharing an experience from a presenter who was 'projected'.
Basically a virtual concert venue for example where process affinity
scheduling took into account where you "were" in the 3D space. The questions
are many and as far as I can tell the folks who research this are few and far
between.

Process migration continues to be a hard thing to do. VMware and others have
made progress, virtualized interfaces and peripherals, self contained 'state'.
Works 'ok' at the VM level but still doesn't work at all AFAIK at the 'thread'
level.

World of Warcraft addresses this somewhat by 'instances' which is that under
certain circumstances (entering a dungeon for example) you and your party of
5, 10, 25, or 40 people all transition (through a loading screen) into a place
with nobody else (except you). They can dedicate machines to host an instance
and do process migration at that level.

DARPA is interested of course because they would love to have a way of
connecting warfighters into a virtual command and control center regardless of
where they are physically. Essentially being in two places at once is a big
force multiplier.

------
EarthLaunch
This was particularly interesting for me. I've already built a scalable MMO
server architecture in Erlang, which hosts all players on one world, using
server nodes that can be moved between machines. I calculate that my server
architecture can handle up to 10,000 moving players in a local area. That's
characters, not EVE-style ships (characters are heavier).

If anyone is interested in talking about this, helping, or funding me, please
get in touch. My server is called Proxima and my game is First Earth (you can
find it if you're interested). I've been slowly bootstrapping it for 3 years.

~~~
outworlder
Have you got more details? Links?

~~~
EarthLaunch
I haven't been public about my process of building the server or its details
(I just don't have time), but the game is at: <http://firstearthgame.com>

Someday I'll make time to blog about the server.

~~~
bestes
Wow, those screenshots look impressive!

------
DigitalSea
I always wondered how Eve coped with what seems like a massively data heavy
game, keeping track of battles and logging data especially for epic battles
where alliances are pitting their expensive ships against each other. Pretty
insightful article, would love to know a bit more about their development
processes and how they go about pushing new changes up. It's such a sensitive
game, changing just one thing could throw the whole game into disarray. It's
crazy when you think that they not only have to track battles, but they also
have to capture every other action and movement within the game; mining, agent
missions, the marketplace and all of the other components that rely on
accurate and up-to-date data capture and realtime reporting.

What makes this all even more crazier is the game has been around for over 10
years and counting. I still play Eve everyday, once you get past the menial
tasks, you can have some real fun with this game being a drug runner, miner or
just straight up renegade roaming the infinite environment.

------
Draiken
The environment EVE creates in almost every aspect of the game is just
fascinating.

It is the only game I ever played I actually felt an enormous thrill and
adrenaline rush when playing pvp. It doesn't matter if you're pvping alone or
with a fleet. The fact that you actually lose/gain on a fight differentiates
the game from any other MMO. Events that you may have experienced in-game can
be recorded forever and actually impacts the whole game universe.

I really hope more companies invest into having one persistent world in their
MMOs. Instead we see this segregated servers in which great
players/achievements/events are limited to this little server. It ends up
lessening the interest of many players, specially older ones.

Funny thing is that, normally, when the hit MMO stops being a hit they end up
merging lots of servers anyways. Which IMHO end up lessening even more the
time people have spent on that game.

~~~
Scramblejams
"The fact that you actually lose/gain on a fight differentiates the game from
any other MMO."

Could you please expand on this? I'm interested in this point, but I haven't
played Eve or other MMOs.

~~~
guizzy
For instance; I play Guild Wars 2. If I die, I lose a little bit of in-game
cash from having to repair my armor after restarting (less cash than I can
make in 5 minutes), and will possibly have to have to trek a couple of minutes
back to where I was.

But back when I played EVE (that's a very long time ago, the game's changed
since)... I was playing as a trucker in low-sec, essentially carrying goods in
somewhat dangerous territory. I was really lusting for T2 transports (blinged
out trucks), and it was my goal to get one. So I trained and saved for months,
until I finally could afford it. I'm not a very patient person, so I took it
for a ride before it was fully fitted with the modules I needed for my
protection. Some pirate jerk blew it up. It was insured, but I wasn't quite
able to easily replace the modules I had lost. In one fight, weeks of my in-
game time were lost.

In other MMOs, winning and losing can mean that you've wasted a few minutes of
your time. In EVE, it can mean months of wasted effort.

~~~
Scramblejams
Thanks for your reply. Do you think this game mechanic would work without the
availability of insurance? I would think all but the most hardcore would be
dissuaded.

~~~
Draiken
Well, insurance in EVE does not free you up to suicide your ship. There is a
saying in EVE "don't fly what you can't afford to lose at least 3 times". It
is a hardcore game, for those who want it to be :)

You can always just live in high-sec (safe zone) and probably never lose your
ship. Of course the game tends to reward you for taking risks.

------
plasma
Second Life (I may be wrong on the game itself) talked about how it manages
game scale across many servers.

The entire game is a virtual machine.

When a player leaves a server boundary, their player VM state is packaged up
and sent to the next world server and then resumed on that server.

Players don't even notice they crossed server boundaries, and this was
possible because the game can be paused mid flight for that player - pretty
neat.

------
clarky07
I haven't played EVE in years, but I always love reading these reports to
reminisce. It took a huge time investment but it was a lot of fun. Reading
that CFC apparently has hundreds of titans and lost 3 just in this battle is
crazy. I was playing when the first Titan was built, and I remember distinctly
when the first one was destroyed.

------
mseebach
On moving the game to a beefier server: _“This is the machine that systems get
reinforced on when players request that. Unfortunately, the same thing above
applies – anyone in the system when the move happens gets disconnected.
Because of this, it’s basically never done,”_

It seems like a fairly small addition to the game logic to allow the game to
freeze for a few seconds and be forcibly moved to another server. Is this kind
of scenario so rare that it's not worth the trouble, or am I missing something
else?

~~~
CJefferson
The problem (I believe) is that they don't have a way to 'freeze' and transfer
a running game. Think of it like any program -- it is easy to "save your work"
and load the file on another computer. It's much harder to exactly save the
state of all parts of a running program and transfer that to another computer.
Each user is represented by a stackless python program, these are hard to shut
down and move.

As you say, it could probably be done, but it would be a lot of work, which
wouldn't get used very often.

~~~
archon
The other problem is that they would have to 'pause' the entire game while the
physical move was in progress, otherwise players would still be able to move
around in the rest of the game world (other than the cluster of stars they
took offline to move them). As I understand it, this is already sort of a
problem with the Time Dilation mechanic they currently use; time moves at 10%
normal within the fight, giving people outside the slowed-down star systems
time to deploy forces at 10x speed relative to the fight.

------
warmwaffles
And of course it's a video of the former Goonswarm[GOONS] now [GEWNS]

------
Syssiphus
Aww, and now I want to start playing EVE again :).

Great article, great battle.

------
OGC
This alliance has some guy who does simultaneous translation to russian?

------
dholowiski
tl/dr: Sharding and slowing everything down.

~~~
lflux
Eve isn't sharded in the common MMO sense.

~~~
dholowiski
No, but they talked about disconnecting the battle from the surrounding solar
system, and the negative consequences. Essentially sharding.

~~~
lacksconfidence
Actually they do not disconnect the battle from the solar system the battle is
in.

What they said is they run many solar systems on one server. During a major
battle they migrate those other solar systems to different servers so that the
server only has to deal with the one system.

So in a sense, yes, each solar system is its own shard.

