Hacker News new | past | comments | ask | show | jobs | submit login
How the EVE Online Servers Deal with a 3,000 Person Battle (penny-arcade.com)
331 points by JGM564 on Jan 29, 2013 | hide | past | web | favorite | 131 comments

The report of the battle they link to (http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/) is even more fascinating to me. You spend hours and hours of effort in order to get a ship, and one way of thinking about them is as the time spent to earn them compressed into 3d form. Unlike (most) other MMO's, battles actually destroy things. So in that youtube clip, you're watching years and years of effort getting evaporated because of a mis-click.

In most of the games I play, the pay off of the grinding is getting to enjoy fighting more. But I could never play EVE, because battles would be so anxiety provoking. There would be no pleasure at all in earning a Titan, because I would be forever terrified of losing it.

Of course, all games are kind of there to destroy your time. But for some reason it doesn't feel quite as wasted when you have a virtual thing there that is bad-ass in proportion to your time investment.

Funnily enough, losing a ship doesn't really make me feel like I've just wasted hours and hours in that way. The real "investment" isn't the time converted into money through ingame activities, be it generating money through fighting NPC pirates or playing the market, it's the time you spent getting into a position where your characters can effectively make money for you, and that isn't so easily taken away from you again.

I don't really think about a pvp ship much differently than about ammunition. They're an expendable resource, and you probably have a hangar full of replacements just waiting for you to wake up in your cloning pod. It's probably a bit more involved with capital ships and the like, as they are really a group effort so the replacement capacities might belong to your alliance and not you personally, but they're still accounted for before a shot is fired.

In a way, this is really what makes EVE interesting. Losses needing to be replaced doesn't only make EVE battles meaningful on a different level than matches in other games (though I hesitate to say more meaningful...), but it also allows the internet spaceship economy to be very central to the game rather than just some mini-game-like distraction. "Item creation" becomes a question of securing resources and production pipelines and supply lines and whatnot, rather than a one-off effort that precedes the "actual" game.

That feeling of "My ship! My beautiful, beautiful ship!" is offset by the fact that major alliances tend to have strong reimbursement programs. If your dreadnought goes boom in a fleet op, you get the cost of another hull and gear, and get to draw on corp/alliance stores as well.

Otherwise, they'd never get their supercap pilots out of the hanger :)

If you don't have insurance on your Titan, you shouldn't have a Titan.

What would be the point of having all this cool stuff if no one else even gets to see it and you don't get to use it?

If you spent six months grinding the ISK to buy and outfit a carrier, how willing would you be to take it into a battle where any of a hundred things could destroy it, including server lag or a random disconnection?

I would invert the causality. Why would I spend six months outfitting a carrier if I had no intention of using it in combat or even flying it anywhere where other people would see it?

Same reason people spend hours tending their farmville farms...

I've never played Farmville, so I probably shouldn't speculate, but I'm under the impression that it is at least ostensibly a social game where you invite people to help with or at least view your farm.

Yeah, but so is EVE and other games like it. I have never played EVE for any significant amount of time, but I did play WoW for awhile and there were tons of things that people would sink time into for purely social reasons. I imagine EVE is the same way. You don't just want a Titan because it is useful in the game, you want a Titan so you can have a Titan.

It's more of a social ponzi scheme. The fun part is the very-passive, easy progress. You just click on things. When you get people to participate in your farm, you're just saving yourself lots of time and clicks.

Well, you wouldn't do it at all. That's why the alliances try to help their members after fights.

Maybe you just want to use it to pick off all the smaller guys. You enjoy driving it around and being the Big Man On Campus. But, you don't want to take it into a real fight, you might avoid an actual conflict with it.

Whipping it out when there's a major battle going on, becoming the star and centerpiece (or one of them) of it, :p.

Strategy isn't just about throwing everything out as soon as it's created and ready to go pew-pew.

In the words of the Lord, our Saviour, Mr Robert Downey JR...

"Peace means having a bigger stick than the other guy"

Alliances accumulate supercapital fleets so they can fly their collections around with relative impunity. Of course it tends to be awful for everyone who doesn't get to be in the big fleet - as they rarely get to play. This leads to the congregation of supercapital pilots in the same alliances.

Pandemic Legion is famous for this - as they have long had the biggest fleet, used it the most, and thus attracted large numbers of other supercap pilots. But they do not hold space.

And to correct a seemingly prevalent misconception - most titans and supercarriers are personal possessions. There are some alliance/corp owned ones - but the majority are simply owned by long time players who either botted (completely un-policed for a very very long time), scammed, traded, speculated, Wormhole'd or Ratted/missioned (Oh god please no - boring monotonous PvE in the extreme for low returns) their way to 80b isk fortunes.

One thing to note, there is a major typo which makes the history between these two coalitions confusing.

Original > Years before, the Test Alliance was part of the HoneyBadgers in a hulking super-coalition. Seeking to carve out a piece of the galaxy for its own, a large portion of Test broke away from the accord to form HoneyBadgers, an independently operating group still pledging allegiance to Test but not to GoonSwarm.

Years before, TEST was part of the Clusters (really: Clusterfuck Coalition, aka CFC). Only in the past year did TEST split off from the CFC and form the Honeybadger Coalition (HBC). TEST and Goons are old friends, so it will be very interesting to see how the current hostilities shake out.

Nobody actually "earns" titans. It's simply not something that an individual would ever fly. Titans belong to the biggest corporations, alliances, and coalitions, and are essentially floating cities that support fleets of other ships. They're a strategic resource that individual players would never use.

In the case of this particular battle, CFC lost 3 titans. This alone is several thousand dollars worth of real-world money, but it will barely be noticed. CFC has hundreds of titans, and the pilots who lost ships will simply have them replaced by their corps. So no one actually lost anything; the coalition will have to drag a few titans and a few dozen supercapital ships out of its reserves to re-arm its pilots.

> They're a strategic resource that individual players would never use.

Except for that year or two when they could shoot smaller ships with immunity, a LOT of people in nullsec alliances got one with their own funds just to shoot bigger guns with impunity.

If they have so many Titans, why not bring out ten of them and sweep the battle?

They already had more than 10 titans. There were almost 3000 ships in the battle at its height. Also, this battle is a bit different in that it happened in low-sec space, where certain rules are enforced by the game's NPC Empire. So you can't use the Titan's most powerful weapon, which makes it vulnerable. Conversely, you can't use area denial interdictors, so many of the CFC's ships were able to escape when it became clear that they were doomed.

Also, travel takes a long time. This battle lasted 2.5 hours and both sides scrambled everyone, waking people up at 4am to go fight. The coalition headed by reddit arrived with 500 ships just as the battle was ending. Part of the reason the folks from Something Awful got beaten as bad as they did is that many different alliances, not just TEST, showed up just for a chance to stick it to them.

Excuse my complete ignorance but I can't resist.

Why was everyone trying to stick it to Something Awful ?

They're the most powerful faction and have been for ages. They've also mastered metagaming and have pioneered some new, truly evil tactics.

For example, Sovereignty Wars in EVE (taking territory from others) is a long, gruelling process that can go on for months. So the goons started launching their attacks at 3 or 4 AM on a monday, to force their adversaries to wake up and go into work with no sleep. They'd do this daily, for a month. In the end, rather than defeating their foe militarily, the opponent would get demoralized and give up, surrendering all of their territory rather than have EVE continue to affect their real lives.

Beyond that, though, they're simply an incredibly powerful faction. There's a resource called technetium that's needed in order to build any of the massive ships, so it's vital for any sizeable corp to have. The Goons grabbed a virtual monopoly on it, and formed OTEC which is a cartel whose goal is to fix the prices of technetium to make it so other groups could never match the goon's strength in massive ships, while growing ridiculously wealthy in the meantime.

Basically, the goons are just really shitty to play against. Couple that with the fact that they and reddit (who used to be the closest of allies) own about 2/3s of everything there is to own in the game, and many players really, really want them to go to war and kill each other off.

All's fair in love and Eve, :p.

In a case like this, it seems the non-SA folk got fed up and ganged up on them. As it should be. No dominant force can remain dominant; if they piss off enough folk, either by being evil or just by owning all the things, said folk will band together and take them on.

And that's one part I love about Eve (despite not playing it); the dynamic and self-organizing nature of the universe and its market.

I played Eve prior to Goonfleet, and I get a similar impression: they're an entity run by adults who are smart and find meanness hilarious, instead of the usual Eve alliance management of European teenagers.

Hopefully someone can give you a better answer, but you might have heard of 4chan? Something Awful is where they came from.

They're actually a major cultural force in EVE. ("Shoot blues" "Little bees") They're a bit like America in international relations, except there are fewer redeeming qualities.

Something Awful (~1999) was around way before 4chan (2003)

http://en.wikipedia.org/wiki/Something_Awful http://en.wikipedia.org/wiki/4chan

That's what he's saying. 4Chan came from SomethingAwful

>you might have heard of 4chan? Something Awful is where they came from.

It's actually more the other way around.

I'm not very familiar with either site's history, but Something Awful was around for a few years before 4chan.

You're right of course, and that's actually what I meant to say. This'll teach me to post late at night.

4chan might literally be "something awful", but the site Something Awful came FROM 4chan. As did half the "meme" sites on the internet, unfortunately...

Just reinforcing JonnieCache's point.

EDIT: Apparently SA was around before 4chan. grr. In any case, they are fairly independent cultures. One charges $10 to register, the other doesn't require a username/login...

Actually, 4chan was founded by a SomethingAwful member, moot, who I think frequented the anime forum on SA (ADTRW). I'm not sure where this reversed order of formation came about.

Nowadays of course SA generally stays away from 4chan, which has become about a lot more than anime and games.

Goons are pretty much well-known for being trolls in every game. Sort of their trademark to be annoying, so... :)

They play to have a fun laugh and how much you expend is upto you. Like any big entity at the top then they get haters and fans alike. But they do it basicly for shits and giggles and don't roleplay the honour card unless it suits them at the time for a laugh.

Because Titans are shitty fighting vessels. They are essentially portable warp gates with fireworks. And they are expensive.

Not just that. You will train over a YEAR to fly a titan, and by flying I mean not just be able to undock it but actually be useful in it. The books costs hundreds of millions to learn so Titan is not a solo adventure it's is always a corporation's operation.

Oh, and since you cannot dock Titan (it won't go inside the station) you will need to park it somewhere, most likely near POS. And if you plan to do something else with your pilot - you need to have TWO pilots to be able to pilot Titan because once you exit ship - someone can steal it. So you need an alt who will be always sitting in the Titan when you log off :)

That's why ppl fly blops in cruisers and battlecruisers. Fun. Fast. Cheap.

One of EVE's sayings is "Don't fly what you can't lose."

In general, you get that kind of ship especially for these types of fights. There isn't much base-building or PvE inside of EVE; it's all about the PvP. Being able to say that you flew a Titan into a battle is a major accomplishment inside of the EVE universe, so you don't care as much if you lose it (of course, you want to keep it safe, but only to fly it into more battles).

Right on the money, but titans are also arguably most valuable for logistics - "jumping" large amounts of ships over large distances. Some may never see battle because the corp or alliance needs them for this purpose and doesn't want to risk losing them.

(Unless, that is, the pilot presses the wrong button and jumps the titan itself into battle, rather than the fleet it was trying to move. Which is what started this whole fight.)

Here's a much more in depth report of the battle. http://themittani.com/news/asakai-aftermath-all-over-cobalt-...

It's a very fascinating read.

It's a lot better read than the PCGamer article, considering that it points out the negative metagame aspects of Time Dilation (TiDi). Part of what made that battle a slowly increasing shitstorm was the fact that reinforcements could arrive much faster than combat could resolve.

That means that what socially could be called a flash crowd accretes in that sector and drags the servers further and further down.

Except I don't understand a single word of it.

The risk element, to me, makes EVE a much more fun game than many others when it comes to combat. The pay off in fun, excitement and stories to be told is so much greater when there was a risk involved.

Kinda like real life...

To be fair, most people who are using the high end ships aren't getting their funding by grinding anymore.

Its much easier to play the market and/or prey on new players moving expensive stuff in areas they think are safe.

Incidentally, probably the most fascinating part of Eve is really the economy. It's by the closest thing to a 'real' market ever created in a virtual setting.

Most income for large alliances comes from mining moons and from taxes on members' income. Ships lost in these alliance-level battles are usually (at least partially) reimbursed by the alliance.

Incidentally, this is what makes some territory more valuable to hold for an alliance, and worth fighting over - some areas can generate much more income than others. Most of the high-level metagame happens over who controls what moons, which is actually invisible to the average player.

It risk vs reward. To own the resources that generate large amounts of ISK/Resources, you'll need to risk your ships to defend it. You can grind all day long in places which has little to no risk to your ship, but it will take a long time to earn enough resources for a nice ship.

In general, this is actually a major problem with EVE right now. The risk vs reward balance is off with the game at least in the sense of an individual player. About the only thing this applies to is alliance income from moon minerals and market speculation.

Most alliances who own space in EVE though don't actually hold the moons that are valuable and so its pretty much all risk for them and no reward at least in terms of money, you do get a reward of having fun.

For the player as I said the risk vs reward ratio is very off balance because there is really no advantage to living in an area that you are more likely to be killed or participating in any pvp of your own choice

Heh, I could never play EVE because stuff like this would make it more addicting for me.

In EVE, you have to think of your ships as ammo.

[Edit: this comment turned out not to really be about Eve and more about the difficulty of real time MMOs]

This blows my mind. I've been thinking about writing a small multiplayer game for a few weeks. And today I was actually doing back of the napkin calculations. The thing that quickly became clear is bandwidth is O(n^2) where n is the number of players in the same location, since you have to share the location and velocity of each player with each other player.

Here's an example of a calculation I was using. It's the monthly bandwidth in TB of having a certain amount of players in a shared space (3000 in this case), assuming that the location/orientation/velocity of each player's ship is contained in a measly 48 bytes and you attempt 30fps.

(48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB

THIRTY THREE PETA BYTES to simulate a single battle for a month.

I'm looking around at various VPS providers and the bandwidth they offer. Using the naive calculation above and the cheapest Linode offering and it'd take $3 million/month to support a single one of these ongoing battles.

Now, I'm sure the numbers above aren't used in reality. Obviously, you can reduce the updates per second... but not by much. You can't shave much off how much you send. It at least gives you a sense of the orders of magnitude required to support such a thing. Even contemplating supporting 100-1000 people at once is looking very difficult for me to pull off.

Hey, I used to be a games networking programmer, and your estimate is pretty good. A few corrections that bring it down to something not entirely unaffordable:

* You can bit-pack your structures a lot more efficiently - let's say 2 bytes for a local player ID, 10 bytes for each of velocity/orientation (quaternion represention and heavily quantizing the theta component), and 10 bytes for the location. Clients are never allowed to determine the canonical physics simulation, so all of the above are really just for display purposes, and can be trimmed down as appopriate - we don't need to worry about desyncing (as I'll explain in a bit)

* 30 packets-per-second is way too high for network play, with a game of EVE's mechanics you could probably get away with something perhaps as low as 1 pps. Intermediate simulation of the player entities is done by a technique called dead reckoning that's linked earlier (though in practice you'd use a slight improvement on it to stop entities leaping around the world).

* Sometimes game mechanics allow you to strongly cut down the number packets you send. For example, in EVE, it might be desirable to not send information about a ship if it's completely occluded by another ship, or to only send information about ships that are in a frontal cone ahead of you. This usually doesn't affect the bandwidth function (though sometimes it does), but you can almost always cut a constant factor of (rule of thumb) 50% off the bill.

So that ends up being 32 * 1 * 3000 * 3000 ~= 250MB/s, which looks about right to me. One thing you didn't account for is that you don't typically allow MMO game clients to connect to each other, but to multiplex everything through a server. So it ends up being twice as large as that - 500MB/s.

Good luck with your game! 100 interacting players in an MMO is a challenging target but not an unreachable one for a single-developer game, and it's definitely a very interesting project to undertake.

Eve is, canonically, 1 cycle/second; That is, the server ticks the game state once per second. Some modules have finer-grained cycle time than that, but their extra is simply saved up tick to tick.

I'm a game developer bootstrapping an Erlang-based game server that attempts this.

You don't have to send 48 bytes each frame (you can usually send only delta), and 30/sec is only necessary in a reaction-based FPS type of game. Hosting players in a local area, local meaning they can see each other and interact, is n^2. My calculations a year ago came up with a per-minute cost for the amount of AWS hardware it would require to host 1,000 and 10,000 'local' players. Basically, it wouldn't be affordable to have a huge gathering/battle more than a few times a month. It would be VERY expensive. Plus, each player has to have enough download bandwidth.

This type of thing will definitely happen in the future. Clustering is still in its infancy, and it's picking up steam. And there are smart optimizations to be made, such as sending more frequent data about nearer players. You can also design around it by not making players usually local to each other even though there is a single seamless game world. I'm doing that.

Is AWS feasible for large scale game hosting? I would have thought co-location would be much more economical in this case.

AWS has lower upfront cost, doesn't require hardware management, and it's easier to provision consistent servers during growth or temporary peaks. Ideal for an unstable clustered system being run by a startup.

I'm mainly talking about EC2, since some other offerings aren't suitable for responsive games. I tried a SimpleDB backend to be hip a couple years back, but it has latency and designed failure rates that are impossible.

I haven't done the math but I'm guessing that with a popular game the costs of hardware and hiring someone to do server management would be wiped out by EC2 costs pretty quickly.

Not sure how spikey game traffic is, but it would seem unlikely that your game is super popular for a day and then drops right off the next as would be the case for many websites.

For example if your game business model is charging some nominal fee per month (say $10) you might find that heavy players can burn through way more than $10 worth of EC2 traffic in a month.

Anyone know if any popular games are hosted on EC2?

Doesn't Sony Japan or Amiga Japan use EC2 for their Playstation 3 servers?

Not sure, though I think the Sony servers are simply used for matchmaking and authentication services and do not actually host most of the games themselves, that is delegated to consoles.

Most RTS games, including Starcraft 2, only update the simulation every 100ms or more. Older RTS games like SC1 and AOE2 used "ticks" of 200-250ms. Essentially, every "tick", player input would be received by the server and distributed back to players after validation for processing or display on the next tick (depending on whether processing is done server-side or client-side). The lag is hidden by visual or auditory cues acknowledging the input (i.e., "Yes, sir!" or "For the Horde!").

This greatly reduces the bandwidth requirements because you have extra time to transmit the data to the user; you only send some of the unit data in each packet. Moreover, you can limit the data to units within X radius of the player/camera and simulate any ships outside of this radius, transmitting only major events like the destruction of a ship. (By simulate, I mean the game could just create random ships and explosions client-side to give the effect of a major battle without actually transmitting or receiving data about those ships; they would be eye-candy only.)

There's some pretty detailed information about the Unreal Engine [1] that might give you some ideas to think about around network protocols for games. At the time they were dealing with 28.8k modems, so there are quite a few tricks in there about saving bandwidth and prioritising some actions over others. [1] http://udn.epicgames.com/Three/NetworkingOverview.html

I think one thing Eve has going for it (and this is only an assumption) is that it wouldn't need to sync each player's position on each frame. It could reduce the load dramatically by only sending the acceleration / orientation data for a ship.

So if a ship has set a course and has stopped accelerating it doesn't need to send anything else unless that course is changed.

This is exactly how these types of problems are solved. Local clients run simulations based on known locations/directions, and are synced with the central server at a specified interval.

This allows a client to have a light-weight "cached" view of the reality, and the only trade-off is the occasional discrepancy from the reality. However, if you send updates about changes in speed/directions in real time, you get a pretty reliable representation of the state known by the server.

Depending on the game (it's been 10 years for me) one 'hack' was to head in one direction, pull the network connection for 3-4 seconds, head in the opposite direction, and plug back in. This allowed you to escape or hide in the terrain fairly reliably.

With higher bandwidths and server capacities I'm guessing these timeouts have been reduced, but never underestimate the player's ability to abuse your trade-offs.

There was an exploit in eve sort of like this. What you did was start warping toward a planet on the far side of the solar system and then kill the application as you got close to it.

The effect was to allow you to sling shot past the destination at warp speed, and if you logged back in at teh right time (seconds later) you could setup a waypoint far outside the outermost celestial body.

(Normally you can only warp to planets and moons, etc... you can't just pick an arbitrary direction and engage warp).

That problem is not because of the trade off. The problem is that the server believe the player's client when it provided an updated location. The 'correct' behavior in this case would be that the server resyncs the client, and the player observe's his ship jump to where the server thinks it should be.

Sure. It's commonly called dead reckoning as I understand it: http://en.wikipedia.org/wiki/Dead_reckoning#Dead_reckoning_f...

But even if you only send a "real" update once a second, I still don't see how this thing scales (that still leaves 1 PB in the above calculation). And if you send it less than that, I imagine things would start to look rather jittery. The "occasional discrepancy form reality" would be awful.

It depends. If you have 3000 clients and a full sync of all visible ships occurs every second, you get 3000 * 48 = approx 150kb/sec/client. Yes, that means that the server must be able to handle 3000 times that speed, but that still leaves us at about 0.5gbit. Yes, it's a very high load, but then again, these types of situations are extraordinary.

Last but not least, compression is your friend.

And in Eve, this is even more feasible because, unlike an FPS or even MMO's like WoW, movement is generally slow and controlled. You don't have direct control over your ship; instead, you can only issue commands like 'head for this waypoint', 'orbit this point of interest', etcetera. This would, I think, greatly simplify server communication of movement and such, compared to direct user input where they could change their orientation much more suddenly.

I haven't really played Eve, so I can't say. I'm imagining ships flying around and changing course constantly in a battle, but if someone can give a better description, I'd appreciate it. It'd be further reduced by several players sharing a ship... no idea how common that is.

What I'm describing is definitely leaning more towards players that are constantly moving around though. It helps explain to me why in Asheron's Call, for instance, when you had too many people in the same city a "portal storm" arose and people started getting teleported out of the city randomly.

Eve doesn't have WASD controls, there really isn't constant input.

The ship movement command in Eve are limited to:

* Set course / speed (double clicking in space, speed set via clicking a speed dial thing). This is not something that gets constant tweaks, more a 'move in a general direction' command.

* Orbit x @ y distance

* Keep x @ y distance

* Approach x

In addition other actions in eve are relatively long lived, there isn't fps style aiming but 'locking on' and activating modules which have cycles times of 2-60 seconds.

In practice, even in the heat of battle, I doubt the average player gets even close to 1 input action per second over the course of a fight.

Combat in Eve is more D&D and less Wing commander. When I played (a few years ago) you would get text based messages pop up saying stuff like "X hits Y for 38 damage"

Eve's gameplay could be represented as a text based game, it is not a fast paced space shooting game.

I also have been toying around with trying to create a MMO fast action space simulation of my own creation. I have a few ideas on how to limit areas of dense players with game mechanics. I haven't even really started on multiplayer beyond the ability to download modify and upload solar systems though.

I think for some really hard engineering problems like this the best course could be to try and find a 'good enough' approach that will allow a significant ammount of players to interact while limiting the upper bound with a game mechanic to make it feel less artificial.


here are a couple videos from the never ending pyopencl port.

http://www.youtube.com/watch?v=lnOmy1ly6M0 http://www.youtube.com/watch?v=XCvRBHtPbzE

If you check the video you'll see it's very slow paced and the ships basically don't seem to move at all (maybe tiny unseen ones do I don't know)


Part of the reason it is so slow is that (as mentioned in the article) the server slows time down (as far as 10% of normal) in order to ensure that nothing gets lost.

My guess is they only sync location on transaction (when you fire upon/are fired upon).

And even that can be optimized away - after all, in Eve, you basically issue commands like 'lock onto this ship' and 'activate this cannon', which will then fire every X seconds - all each client has to know is the target, ship information, and a toggle command for a given weapon, the rest can then be determined completely on the client-side when it comes to the visuals. Every X ticks, data (like health) of the target ship can be sent through - although I reckon that too would be optimized away and only sent on-demand, i.e. when the user actually selects the ship under attack.

Can't you do a P2P system when a lot of users are interconnected and then send some information to the prime server?

The server controls everything. Clients just display whatever the server tells them to (plus maybe some temporal interpolation).

If you start doing P2P, you're opening yourself to all kinds of hacks, and increasing the amount of duplicate data traveling around.

So if you're in a battle of 3000 people, you would have to somehow connect to 3000 people, send 3000 times more data. This would probably saturate your uplink and get you banned by your ISP as a suspected botnet victim.

Fun facts: This fight was between the Honeybadger Coalition (HBC) and the Clusterfuck Coalition (CFC). The core of the Honeybadger Coalition is TEST Alliance, and the core of TEST Alliance is Dreddit, the "original" Reddit-based corp (there are / have been several, some of which predate Dreddit and have since folded).

HBC routed the CFC, which has Goonswarm as its core, which is organized around the Something Awful forums.

TEST Alliance was nurtured by the CFC for a long time, splitting off only in the past year to forge a more independent identity. As such, the hostilities between these old friends is hard to gauge. At this point, the presumption is that it's all in good fun. But that may change.

I was looking for information on this too, because last I heard, TEST and Goons were closely-allied buddies, and here they are fighting each other in a relatively large way. Then again, I think Pandemic Legion and Goons used to work together against BoB, but then seemingly broke ties (and now PL+TEST, as part of HBC, have clashed with Goons). This article clarified things for me somewhat:


As far as I understand the article, the CFC and HBC are the two biggest coalitions in the galaxy, and war between them is a risky bet for either in the current conditions. There's not enough incentive for them to do it just yet, and old ties come into play. (Some commenters mentioned technetium moons as an underlying factor - very profitable sources of income which PL stands to lose in a war.)

From my outsider point of view, the political situation in the EVE galaxy has matured: the Allies (CFC) beat the Axis (BoB), some residual fighting has gone on, and everyone's trending either towards a Cold War or a United Nations. The ethos of EVE predisposes alliances to conflict, but there's a lot of territory and months of sleepness nights at stake when supercoalitions decide to go to war. And these supercoalitions used to be (at least in part) allied to each other against older enemies.

The pendulum might swing again, but this time it won't end as easily as BoB crushed ASCN in 3 months.

Admittedly I haven't followed Eve politics for over a year, but I would be very surprised to see Test and Goons get in a "real" fight (e.g. over sovereignty or revenue-generating moons). They both treat the game in the same way (a strange mix of competence, incompetence, and irreverence) and have too much shared history. I suspect the only reason they're even hostile right now is because they have absolutely no external threat and are bored.

Fits with what I read on Reddit when it came up. Was a fun fight and TEST won but no one is really mad at anyone else:)

It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? Oh, I guess it's 100% utilization, not 100% load. As in, they don't spin down servers to save power during off-hours I guess.

The time dilation is a neat solution to the server load problem, but it's sooo annoying as a player. In beta, it was interesting to watch the entire game desync and grind to a halt, but we could still chat and look around. Interesting, but frustrating.

Two main solutions (given that the article mentions that they have huge machines already):-

A) Allow a single solar system to span multiple machines. Very hard, especially if the server software isn't architected for this. Retrofitting this can be nigh on impossible.

B) Have a few huge machines that can be used to host scenarios like this and, more importantly, have a way of migrating users over to the huge machine seemlessly.

The latter can be done but it's tricky, especially if transferring game state between instances of the server is not simple (I'm not talking about transferring the VM itself with something like vMotion). It comes down to:-

1) Being able to make the bigger machine act as a temporary proxy pushing connections data back to the smaller machine.

2) Having a way of telling clients to make a new connection to the bigger machine and, once that connection is made (and the data is being proxied to the smaller machine) cut the connection to the smaller machine. Users see no loss of service or reconnects at all.

3) Once all clients are now being proxied by the bigger machine; pause and transfer the game state from the smaller machine to the big machine and then continue. Obviously it works best if a chunk of state can be transferred in the background and then the final transfer (and pause) is as short as possible in order to transfer over the bang up to the minute state.

Option (A) is always the proverbial "In v2 of the server we'll do it a completely different way..."

Which never seems to come around, because the new hardware is so much faster that it can host what were previously problematic server loads without a problem, and you've got a million other things to write.

Yet players have a tendency to figure out when places are too overcrowded to be fun. So your old problematic load is almost never representative of how many players wanted to be in that area, but merely how many players were willing to put up with that level of degraded performance.

So upon release (or sufficiently close to it to start stress testing, which is conveniently when it's too late to really change architecture) the new limits are quickly hit.

They've been struggling with the issue of multi-thousand player fights for a while now, and have moved towards both of these solutions but are obviously not quite there yet.

For instance, the article actually talks about having said huge machines. There's a way in EVE to inform the GMs about anticipated big fights, at which point they'll do the reinforcement preemptively. In this case, there wasn't such a convenient warning.

I find this post particularly interesting, since what described (outside of doing it at a VM level), somewhat reflect how some Telecom providers build their equipment. Telecoms in North America are properly crazy when it comes to recovering from failure with minimal visible impact to customers.

Usually on the Telecom equipment, the backup / state transfer is done at a process level, not at a VM level as suggested, but it's quite common practice.

The best equipment I've seen, does this by spawning many equivalent processes, and distributing them among the available blades in the chassis. If you have process mgr1, you get a backup1 process on another blade. As mgr1 processes you're call state, it checkpoints all critical data to the backup1 process. If the mgr1 process itself crashes, or the entire blade fails, all the processes are simply re-spawned, contact their corresponding backup process, and transfer all the state information back, and simply resume. Most end users won't even notice. Using this method, I've seen equipment recover well over 30,000 subscriber sessions in under 5 seconds, most of which probably wouldn't even notice, and even if you did it wouldn't be enough to drop you're data connection (VPN, video streaming, or whatever you're doing). We also don't lose you're bill for the usage either ;)

The challenges with applying this to the game environment, is in telecom each user session is independent, and doesn't really interact with other sessions, so we don't have an issue of a single process becoming overloaded and needing to free up resources to handle it. However, it would be properly easy to do within this model, since failure is expect to occur and be recovered from.

As a programmer, you have to be properly diligent in the software design, what get's check pointed, when does it occur. I couldn't even imagine trying to retroactively apply this type of design to "legacy" software, that wasn't build from the ground up with this model in mind.

Option 4C) Create a raspberry pi cluster farm, and host every player with one microserver.

How would the microservers keep up to date with each other fast enough to be useful? "eventual consistency" doesn't work with real time strategy games.

How would they not? Latency between servers would be near zero, and if they all handled point to point transactions while broadcasting their state on each transaction, you're getting all the power you need. We're not talking about complex processing here. It's an RNG coupled with simple health and xyz.

Broadcasting their state to what? All the other nodes? So each "microserver" would still need to handle thousands of connections and data for every player?

The real problem for them that forces them to use time dilation is the fact that a solar system can only run on one core. They have a few machines with a 4.4 Ghz or greater processor I believe for system that are high load but currently they are extremely limited in what they can do hardware wise.

>It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time?

I wasn't sure how to interpret that either.

I've always been a bit surprised that the the backend systems for these MMOs aren't a bit more flexible. Though, EvE did launch nearly 10 years ago and it has never had a huge number of subscribers.

I have no idea whether more recent games have solved these problems.

I have no idea whether more recent games have solved these problems.

Most games solve this problem with completely separate servers and population caps. I know in Guild Wars 2 they actually stop displaying players past a certain number to improve performance. This works fine in calm areas like cities, but in PvP it causes issues with being killed by "invisible" groups that the game fails to load in time.

I may be remembering incorrectly, but last I remember reading CCP was basically in uncharted territory on the tech front as far as EVE Online is concerned. No other game developer has even tried to tackle this problem at the level they have, and I can't think of many applications in the world that could comparable in scale and complexity to what the EVE servers have to deal with.

Edit: To summarize, even being 10 years old, no game has come close to matching it in this context.

Guild Wars 2 is stupid about this. At least on release, they stopped displaying ENEMIES FIRST. So if you're in a big enough group, a couple enemies can walk up and none of you can see them b/c you hit the limit from displaying allies, but they can still see some of you guys and kill you while invisible.

If they prioritized displaying just party members (parties only go up to 5 people) and maybe people on your friends list, and then enemies, it would have worked better.

They supposedly just fixed this in the patch yesterday. As a player who gets owned by unrendered thieves, I'm not convinced. -_- (OTOH, I'm also pretty bad at PvP anyways, so whatever.)

400k+ active players IS a huge number. Specially considering they are in the same logical server. Other MMO's handle this by having copies of the world and splitting the population - which is the same approach used by Ultima Online back in 1997.

So actually, EVE is way ahead of the rest, technology-wise.

>So actually, EVE is way ahead of the rest, technology-wise.

That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO.

In any case, I'm not making a dig at EvE, rather acknowledging that the game has had a long life and has never been a multi-million subscriber behemoth. Therefore, they might make concessions, or allowed the persistence of previous limitations / decisions in design/infrastructure for lack of resources [1][2].

1: http://news.ycombinator.com/item?id=5135873

2: http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onli...

That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO.

The "typical" MMO doesn't even try to handle large populations. The use of separate servers, population caps and login queues are how just about every one else deals with congestion problems.

Even from your second link there is the quote "Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry." Sure that is from 2008, but newer MMOs are built using the same overall server architecture you saw back in the days of Ultima Online.

400k subscribing users is a great big pile of cash to work with.

I wouldn't call over 400,000 players '[not] a huge number'. It's the 2nd or 3rd most popular paid-subscription MMO.

As you can see here http://eve-offline.net/?server=tranquility , there is about 44k people online at this point, probably a bit less since most people have more than one account.

That seems like a fairly substantial amount of users in a single world. Is there any other game world with this many active users in a single world?

The specific, relative number of subscribers is not the point.

That said, I wouldn't call something huge when there are comparables (even if it's just one or two) which dwarf it and others which (at some point) exceeded it.

I can't think of any games that deal with PVP on this scale either.

If memory serves, I believe that they mean that the servers run at 100% capacity, not 100% load ie. CCP does not throttle back system resources during periods of light use.

This is a nice article.

A friend of mine did some research for DARPA/NSF on internet 'crowds.' His research was looking at the question of fractal gatherings, which was basically you were 'near' the people around you in the virtual 3D space and could hear what they were saying, you were adjacent to people who were one space away but could hear them if they 'shouted' and the crowd was at a gathering that was sharing an experience from a presenter who was 'projected'. Basically a virtual concert venue for example where process affinity scheduling took into account where you "were" in the 3D space. The questions are many and as far as I can tell the folks who research this are few and far between.

Process migration continues to be a hard thing to do. VMware and others have made progress, virtualized interfaces and peripherals, self contained 'state'. Works 'ok' at the VM level but still doesn't work at all AFAIK at the 'thread' level.

World of Warcraft addresses this somewhat by 'instances' which is that under certain circumstances (entering a dungeon for example) you and your party of 5, 10, 25, or 40 people all transition (through a loading screen) into a place with nobody else (except you). They can dedicate machines to host an instance and do process migration at that level.

DARPA is interested of course because they would love to have a way of connecting warfighters into a virtual command and control center regardless of where they are physically. Essentially being in two places at once is a big force multiplier.

This was particularly interesting for me. I've already built a scalable MMO server architecture in Erlang, which hosts all players on one world, using server nodes that can be moved between machines. I calculate that my server architecture can handle up to 10,000 moving players in a local area. That's characters, not EVE-style ships (characters are heavier).

If anyone is interested in talking about this, helping, or funding me, please get in touch. My server is called Proxima and my game is First Earth (you can find it if you're interested). I've been slowly bootstrapping it for 3 years.

Have you got more details? Links?

I haven't been public about my process of building the server or its details (I just don't have time), but the game is at: http://firstearthgame.com

Someday I'll make time to blog about the server.

Wow, those screenshots look impressive!

I always wondered how Eve coped with what seems like a massively data heavy game, keeping track of battles and logging data especially for epic battles where alliances are pitting their expensive ships against each other. Pretty insightful article, would love to know a bit more about their development processes and how they go about pushing new changes up. It's such a sensitive game, changing just one thing could throw the whole game into disarray. It's crazy when you think that they not only have to track battles, but they also have to capture every other action and movement within the game; mining, agent missions, the marketplace and all of the other components that rely on accurate and up-to-date data capture and realtime reporting.

What makes this all even more crazier is the game has been around for over 10 years and counting. I still play Eve everyday, once you get past the menial tasks, you can have some real fun with this game being a drug runner, miner or just straight up renegade roaming the infinite environment.

The environment EVE creates in almost every aspect of the game is just fascinating.

It is the only game I ever played I actually felt an enormous thrill and adrenaline rush when playing pvp. It doesn't matter if you're pvping alone or with a fleet. The fact that you actually lose/gain on a fight differentiates the game from any other MMO. Events that you may have experienced in-game can be recorded forever and actually impacts the whole game universe.

I really hope more companies invest into having one persistent world in their MMOs. Instead we see this segregated servers in which great players/achievements/events are limited to this little server. It ends up lessening the interest of many players, specially older ones.

Funny thing is that, normally, when the hit MMO stops being a hit they end up merging lots of servers anyways. Which IMHO end up lessening even more the time people have spent on that game.

"The fact that you actually lose/gain on a fight differentiates the game from any other MMO."

Could you please expand on this? I'm interested in this point, but I haven't played Eve or other MMOs.

For instance; I play Guild Wars 2. If I die, I lose a little bit of in-game cash from having to repair my armor after restarting (less cash than I can make in 5 minutes), and will possibly have to have to trek a couple of minutes back to where I was.

But back when I played EVE (that's a very long time ago, the game's changed since)... I was playing as a trucker in low-sec, essentially carrying goods in somewhat dangerous territory. I was really lusting for T2 transports (blinged out trucks), and it was my goal to get one. So I trained and saved for months, until I finally could afford it. I'm not a very patient person, so I took it for a ride before it was fully fitted with the modules I needed for my protection. Some pirate jerk blew it up. It was insured, but I wasn't quite able to easily replace the modules I had lost. In one fight, weeks of my in-game time were lost.

In other MMOs, winning and losing can mean that you've wasted a few minutes of your time. In EVE, it can mean months of wasted effort.

Thanks for your reply. Do you think this game mechanic would work without the availability of insurance? I would think all but the most hardcore would be dissuaded.

Well, insurance in EVE does not free you up to suicide your ship. There is a saying in EVE "don't fly what you can't afford to lose at least 3 times". It is a hardcore game, for those who want it to be :)

You can always just live in high-sec (safe zone) and probably never lose your ship. Of course the game tends to reward you for taking risks.

Second Life (I may be wrong on the game itself) talked about how it manages game scale across many servers.

The entire game is a virtual machine.

When a player leaves a server boundary, their player VM state is packaged up and sent to the next world server and then resumed on that server.

Players don't even notice they crossed server boundaries, and this was possible because the game can be paused mid flight for that player - pretty neat.

I haven't played EVE in years, but I always love reading these reports to reminisce. It took a huge time investment but it was a lot of fun. Reading that CFC apparently has hundreds of titans and lost 3 just in this battle is crazy. I was playing when the first Titan was built, and I remember distinctly when the first one was destroyed.


I might be. You don't have any contact information in your profile.

On moving the game to a beefier server: “This is the machine that systems get reinforced on when players request that. Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected. Because of this, it’s basically never done,”

It seems like a fairly small addition to the game logic to allow the game to freeze for a few seconds and be forcibly moved to another server. Is this kind of scenario so rare that it's not worth the trouble, or am I missing something else?

The problem (I believe) is that they don't have a way to 'freeze' and transfer a running game. Think of it like any program -- it is easy to "save your work" and load the file on another computer. It's much harder to exactly save the state of all parts of a running program and transfer that to another computer. Each user is represented by a stackless python program, these are hard to shut down and move.

As you say, it could probably be done, but it would be a lot of work, which wouldn't get used very often.

The other problem is that they would have to 'pause' the entire game while the physical move was in progress, otherwise players would still be able to move around in the rest of the game world (other than the cluster of stars they took offline to move them). As I understand it, this is already sort of a problem with the Time Dilation mechanic they currently use; time moves at 10% normal within the fight, giving people outside the slowed-down star systems time to deploy forces at 10x speed relative to the fight.

And of course it's a video of the former Goonswarm[GOONS] now [GEWNS]

Aww, and now I want to start playing EVE again :).

Great article, great battle.

This alliance has some guy who does simultaneous translation to russian?

tl/dr: Sharding and slowing everything down.

Eve isn't sharded in the common MMO sense.

No, but they talked about disconnecting the battle from the surrounding solar system, and the negative consequences. Essentially sharding.

Actually they do not disconnect the battle from the solar system the battle is in.

What they said is they run many solar systems on one server. During a major battle they migrate those other solar systems to different servers so that the server only has to deal with the one system.

So in a sense, yes, each solar system is its own shard.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact