
Planning for hypergrowth - nanott

======
dhouston
it depends on where your bottlenecks are. if you're serving static content
(e.g. an image hosting or photo sharing service), amazon S3 is a cheap and
simple way to do so. if you have a big database involved anywhere (e.g.
facebook, which has to keep track of user profiles, mappings between users,
groups, and god knows how many hundreds of other kinds of connections), it's a
lot tougher.

in general, you can "scale up" (buy bigger servers) for a little while, but
eventually you'll have to "scale out" (divide the load across a lot of smaller
servers.) amazon's EC2 service lets you do this relatively easily. it's hardly
plug and play though; you have to figure out how to scale out within your app
and ensure that you can split up your data across multiple servers, which is
often not trivial. for example, even if you can partition your data, what if
one "piece" gets hammered with a disproportionate amount of traffic (i.e. one
particular photo album or whatever gets dugg/slashdotted); what if a server
goes down; what if you need to dynamically add servers; how do you protect
data if a disk completely fails; etc.

here's some food for thought from the sysadmins of livejournal and digg, and
how they dealt with growth.

<http://www.getdropbox.com/u/2/Interesting/servers%2C%20infrastructure/danga.com%20livejournal%20backend.pdf>
<http://www.getdropbox.com/u/2/Interesting/servers%2C%20infrastructure/digg%20sysadmin%20presentation.pdf>

perhaps better than the technical solutions are social solutions to
hypergrowth: you can use invites, beta periods or waiting lists to control
your growth instead of turning people away indefinitely, and also set users'
expectations that things may go wrong and to bear with you. it's really tough
to anticipate on the technical side where things will break if you're adding
hundreds of k users; just look at ilike, which got rocked by f8 signups.

there are more pieces too besides just serving the http requests properly :P
presumably you'll need to provide some support to these people, answer emails,
fix things when they go wrong, have some method of discovering and triaging
problems, etc.

------
nanott
So, how does one plan for massive growth during first few days/weeks? Consider
possible scenario of launching and within first 3 days millions of users are
visting the site daily. Maybe this is an obvious (or nonsensical) guestion -
thought I'd ask anyway.

~~~
r38y
I would say plan for some traffic you can afford, but know of someone that can
help you out if you really get into trouble. It's along the lines of "worry
about that problem when you have it, because that problem is a great one to
have".

~~~
danielha
It's a great situation to be in, but it's not a great problem to have. Worry
about this before you launch because turning away potential users isn't a
goal.

------
nanott
Actually, a related and perhaps more interesting question might be:
considering a viral marketing scenario, how can one compute a practical
upperbound on how quickly website traffic would grow?

