

Why are Facebook, Digg and Twitter So Hard To Scale? - yarapavan
http://highscalability.com/blog/2009/10/13/why-are-facebook-digg-and-twitter-so-hard-to-scale.html

======
dasil003
I think it's laughable to put Twitter and Digg in the same category of
scalability as Facebook.

Maybe things have changed since the last time I visited Digg over 2 years ago,
but the social networking aspects are not very significant. The vast majority
of their hits are practically fully page cacheable.

Twitter at least has an interesting scaling problem, but they don't have any
features and they move at a glacial pace.

Facebook on the other hand has a graph that almost as nasty as Twitter's
(minus the million followers thing), but they have 100 times the features, and
they push new code _every week_.

~~~
timf
Facebook also "has about 30,000 servers supporting its operations, hosts 80
billion photos, and serves up more than 600,000 photos to its users every
second."

(check out <http://www.facebook.com/Engineering?v=wall> )

~~~
yarapavan
Facebook handles 30K machines, 300M active users, 20B photos, & 25TB per day
of logging data. <http://bit.ly/EA3oY>

------
jbert
I wonder if there is value to be had in migrating users to increase social
graph locality.

The probability of (B fr C) is greater than the mean if (A fr B) and (A fr C).
That's useful information.

Off the top of my head, I wonder how an algorithm like:

\- I have N shards

\- pick the top N most-connected users

\- assign them each to a shard

\- assign their immediate friends to the same shard

\- randomly fill in other users

would work.

Possible refinements:

\- if the %age of shared friends between two users in the top N is > X, put
both users in the same shard + add the N+1th user at a new shard-seed

\- chase more than one level of immediacy from the shard-seed users to fill
the shard

\- if a shard is full, don't drop to random allocation for 1st- or 2nd- level
friends, but instead put them all onto shard+1

The idea here is that for pull _or_ push you win if you need to contact fewer
shards. i.e. the queries and updates needn't be per-user but per-shard. i.e.
you can query/update for all users on a shard in one sql statement.

If you somehow manage to keep all of ashton's friends on 10 shards instead of
100, then that's a big win, surely?

------
fizx
I've wondered for a while about some hybrid push/pull distribution scheme. I
have 100 followers, just push. Oprah has 1M, 1% of which are currently active,
pull. Sounds complicated, but feasible.

~~~
bsaunder
But is also has to do with how frequently the information is changed/accessed.

e.g. If Oprah only publishes 3 tweets a day, but her one million followers
each check 100 times a day (just to be on the bleeding edge of gossip), it's
much less effort to push the change.

On the flip side, if you post status changes several times a day but your
followers rarely check (daily/weekly), pull may make more sense.

Although, I suppose that a push inherently requires an update/write, while a
pull is generally a read. Seems like this might need to be taken into
consideration as well.

------
x3m
This is in fact an unsolved problem. Current database systems cannot handle
this level of connectivity. Facebook can't do a "push" because of all the
features they support. It gets extremely complicated if they push complex data
into user's mailbox. And twitter cannot do "pull" so they can avoid putting a
cap on the number friends/followers but then that's why they cannot really add
any feature.

------
gaius
Because those who ignore the pub-sub research IBM did in the 80s are doomed to
re-implement it badly.

~~~
wheels
I'll preface this with the fact that I've not read those papers (but today was
thinking through the concept of write hooks in information graphs), but I'd
assume from the common formation of the pattern that it's only really set up
for one-dimensional publishes.

The problem in large scale information networks, more in the Facebook way than
the Twitter way, is that you you potentially trigger a cascading effect in
information updates if you go pub-sub. Specifically, applications that do
interesting things with social graphs have to go beyond basically doing
message passing.

I naturally look at things from a recommendations angle, but if you've got a
new edge that enters the graph that may affect other edges that are connected
to the end points. Those may in turn affect the edges that are connected to
those nodes and so on. You want to avoid something that effectively becomes a
breadth first traversal of the graph doing updates since that's well, slow, to
put it mildly.

This is why large-scale graph algorithms like PageRank work on constantly
regenerating static matrices rather than doing regeneration of the ranks
dynamically, but that naturally is problematic when you're working on data
sets where the most recent data is the most important and is being generated
at very high rates.

~~~
jfarmer
In some models, sure, but is that true in Facebook's? Does anything go beyond
one degree?

I can't think of anything, really. If X and Y becomes friends, X's friends and
Y's friends see it in their feed ("inbox"), but nobody else. Same goes for
tagging someone in a photo, etc.

~~~
wheels
Ad targeting, what's hot in your network, friend recommenders, etc. are all
essentially ranking problems. They're moving more in that direction,
definitely.

~~~
jfarmer
Word. True. I knew there was something I wasn't thinking of.

------
bsaunder
I think the trade off between push/pull is influenced by the ratio of the
'velocity' of data (how frequently it changes) to its access frequency.

If accessed frequently, but changes less frequently, then push makes sense, if
changes frequently, but accessed infrequently, then pull makes sense.

It also seems likely that the same piece of data may have different ratios
from different perspectives.

------
Retric
I spent a few hours mocking up a solution to this type of status updates in
code. It seemed fairly obvious so I assumed it was not a real problem.
However, if there is any interest I can turn it to some sort of blog post next
weekend.

~~~
dlsspy
Sure it's obvious until you have hundreds of millions of nodes that each link
to somewhere between 100 and 2,000,000 other nodes that need to be updated
when any given node updates (assume around 5,000 nodes are updating every
second with a power law kind of distribution).

If you think that's obvious, I think you're significantly beyond all of the
people with all of the "hello world" twitter clones out there.

~~~
Retric
It all comes down how well you can slice the process. There is no cheep over
the counter solution to these problems but a little custom code can go a long
way.

5k node updates per second might sound like a problem, but one core of one
machine can easily keep up with that so you can have several copies _and
several views_ of the whole network graph. Public vs. private messages can be
handled separately and then joined before presentation to the user. You can
separate finding which message to display from the message data. You even get
to display dirty reads as long as the data is <2 seconds old it's plenty good
enough.

Ok, describing the solution based on the above insights takes some time and
pictures but does it still sound horrible?

PS: Twitter was forced to morph an architecture built to solve a different
problem into a working solution. That takes time and can be fairly difficult.
But, starting from scratch it's not that bad.

~~~
scott_s
I think you under-estimate the complexity and performance costs of having a
massively distributed data structure. Having N nodes perform computations on a
graph many times the size of N is easy. Maintaining some form of coherency
throughout all N nodes is hard.

The communication costs are significant, and probably similar to N-body
simulations.

~~~
gaius
_Maintaining some form of coherency throughout all N nodes is hard._

Hence the "eventually consistent" model.

~~~
scott_s
That's still hard. There's a trade-off between accuracy and performance.

------
radu_floricica
I've been wondering for a while why we don't see more often systems with a
huge amount of RAM and asynchronous persistent updates, with a more, how
should I put it, computer science-ish architecture inside. MVC works ok, but
does it really mean we _have_ to use it always? Is it so hard to make system
with a TeraByte of RAM? Why even use memcache? Why not make a clear decision
that stuff like user statuses will never even see the inside of a hard disk
and make use of the simplification it brings? Is there a clear reason, or
simply it's not fashionable?

~~~
thomaspaine
User statuses should never touch hard disk? What happens when your server
restarts and everything you were storing in memory disappears?

~~~
spc476
The greylist daemon I wrote (<http://www.x-grey.com/>) keeps everything in
memory, with a checkpoint (everything gets written out to disk) every hour
(the main process forks, and it's the child that does the writing). A similar
approach might be applicable here.

It could very well be a RAM based approach might be viable once memristros
([http://en.wikipedia.org/wiki/Memristor#Potential_application...](http://en.wikipedia.org/wiki/Memristor#Potential_applications))
are used for memory.

------
siculars
scaling is hard. this is why, for the non-technical audience... also the post
ends with "brave new world." yuck.

------
tybris
Pervasive writes?

