

Handling over 300 million events per day, in real time, on a budget - benologist
http://playtomic.com/blog/post/53-handling-over-300-million-ev

======
AlexC04
Ben, this is really interesting. I was under the impression that Playtomic was
a one man operation, but the idea that you're managing 300 million requests a
day boggles my mind. (at my office we've got a team to handle that).

I'll have to go through your article with a fine toothed comb - but that's
really quite a feat.

~~~
benologist
It is and it isn't a one man operation - I'm the only one actually programming
/ managing the servers / creating it all etc, _but_ in the background is my
friend Antonio who helps finance the development and has for a long time now,
and very recently joined is my friend and over the years sometimes boss Brian.

It's not really fair to say 'I' anymore as we ramp things up on the business
side of it and formalize everything.

------
kprobst
Glad to see someone's doing something like this on top of the Microsoft stack.
I was expecting the usual Linux+Ruby+Hadoop+Redis+etc thing. Pretty cool.

~~~
quickpost
Markus Frind is also doing some pretty huge volume on a Microsoft stack with
plentyoffish.com...

[http://plentyoffish.wordpress.com/2006/06/10/microsoft-
aspne...](http://plentyoffish.wordpress.com/2006/06/10/microsoft-
aspnet-20-performance/)

~~~
kprobst
That's impressive. Thanks for the ref.

------
jbaker
Thanks for the article. I have a question. What kind of latency do you
experience using MongoHQ? And maybe you could discuss this part of the system
a little more?

~~~
benologist
The latency gets up to about 60ms+ but it's mitigated by caching a lot of the
data temporarily. MongoHQ are using Amazon so if you're also using AWS it's
going to be negligible.

I use MongoDB for a few components - user-created levels and leaderboards are
the biggest parts. The big advantage is I can let developers attach custom
data to scores/levels without having to even think about it - it slides right
in just like any other property, and it's just _effortless_. The other big
advantage is price - they're a lot cheaper than getting another server.

The leaderboards is the biggest one running at a total of almost 10 gigabytes
because of the indexing.

I use MongoHQ because there's a learning curve with MongoDB that I never have
time to explore properly, originally I ran it myself and even got it sharded
and replicated, but when it falls over (and it did) you really need to know a
lot more than I do about it. Those guys are awesome and I plan on building
more on their service.

I use the Samus C# library for it, I noticed recently there's an official one
now but it didn't exist when I started using it. One of these days I will
upgrade to that.

Here's the leaderboard database stats. Mostly it's just lots of reads, and
only if whichever server doesn't have the requested data in-memory already.

    
    
      btree accesses 	0.47 /sec
      btree miss ratio 	0.2%
      global lock ratio 	0.1%
      indexes	8
      objects 	24,222,154
      op deletes 	0.00 /sec
      op get mores 	0.73 /sec
      op inserts 	0.57 /sec
      op queries 	4.02 /sec
      op updates 	0.02 /sec
      all ops 	0.02 /sec
      size of data 	3.3 GB
      size of index 	6 GB
      size of storage 	3.7 GB
      uptime	4 days
      connections 	128

