
How MySpace Tested Their Live Site with 1 Million Concurrent Users - cmendis
http://highscalability.com/blog/2010/3/4/how-myspace-tested-their-live-site-with-1-million-concurrent.html
======
slyall
I've read this story a few times and I always have a few problems with it.

1\. The total population of New Zealand is 4 million. Expecting 1 million
concurrent users on the service is very optimistic. Also Myspace wasn't that
popular in New Zealand, these days facebook dominates but previous Bebo was
common among young people.

2\. Testing from EC2 instances against Akamai will not simulate load from
NZers. The December 2009 the closest EC2 servers to NZ were in the US so
traffic would have been pushing completely different servers.

3\. A transfer rate of 16Gb/s almost certainly exceeds any spare capacity in
the New Zealand Internet. Also there are less than a dozen Akamai clusters in
NZ, distributed unevenly and probably most don't have more than 1GB/s
connections.

So while it's a nice story, if they were really testing just for NZ they spent
a lot of time and money testing the wrong thing. Perhaps it could be a story
about how inflated marketing expectations cause cost to be incurred by
engineering?

~~~
thadeus_venture
1\. They launched in NZ, but the service was for everyone worldwide.

2\. Akamai might have played a part, but wasn't the main target of the load
test.

3\. See 1.. the MySpace data center is in the states.

------
Aloisius
I used to do this at Napster, albeit with far less computing resources. I
would record traffic to a machine and then play it back at 10-20x realtime to
see where my performance limits were using a few machines. Of course, it
turned out that most of the limits we ran into were hardware related (the max
packets per second Cisco switches advertised was... optimistic).

Of course, I also just tested in production with real users. I'd redirect more
or less users to a particular server to stress it and see where it broke.
Since we controlled the protocol and the client, I could hide any problems
pretty easily. Users didn't seem to care if a for a couple minutes the service
was a little wonky. As Twitter has demonstrated, if people love your service,
they will forgive some minor disruptions from time to time.

At our peak, we were at 2.3 million simultaneous users, but we had nowhere
near the bandwidth requirements of MySpace. It wasn't until recently with the
rise of continuous deployment did I think that testing in production was an ok
thing to do. I did it because I had to.

