

Scaling memcached at Facebook - snprbob86
http://www.facebook.com/note.php?note_id=39391378919

======
iseff
I must say, Facebook does a lot of interesting work. Things that hardcore
engineers would definitely enjoy doing.

But also things I would hate to support.

For instance, I would NEVER want to have to support making a one-off change to
MySQL as Facebook has done, to enable them to open a new data center
([http://www.facebook.com/note.php?note_id=23844338919&id=...](http://www.facebook.com/note.php?note_id=23844338919&id=9445547199&index=6)).
What happens when MySQL is updated? They can no longer just pull the latest
version. They have to pull the source, modify it appropriately, test, and then
deploy. And if there are bugs, things could get very tricky, very fast.

And this memcached scaling seems like another example. Really cool concept,
changing many low-level Linux things. But, really, do they want to be in the
business of supporting those things? I wouldn't.

I realize they have massive scale. And I realize that at massive scale,
standard solutions may not work. But perhaps they should find ways to do
things at a slightly higher level, in a layer they fully control. This may
also allow them to focus more on their core competencies and not waste
precious developer time.

UPDATE: To make matters worse, from a mail on the memcached mailing list about
this post:

<quote>

I think the results speak for themselves, but I don't know that a merge can
actually occur.

The tree the published is entirely unrelated from the trees the rest of us are
working on. There's no common ancestry or even similar directory layout. As
published, it sort of puts us in a position to either reimplement everyone
else's work, or reimplement the facebook work.

If anyone at facebook is listening, is it possible at all to add this work
onto the codebase where everyone else has been working? We've got a lot of bug
fixes and features we'd really like to not throw away here:

<http://github.com/dustin/memcached/tree/rewritten-bin>

</quote>

~~~
staunch
On Facebook or Google scale it makes total sense. They're using 800 servers
instead of 3,200. That's probably $5-$10 million dollars in savings. Easily
enough to pay the salaries of a few developers to maintain this stuff even if
that was their _only_ job.

It does seem like it's just flimsy excuses not to merge up the changes, but
hopefully they'll put more emphasis on it after a while.

~~~
retyred
you can't put fb and google in the same category. fb is still surviving on
investment capital and frankly could end up in a serious cashcrunch. google
has a blank check for hardware. both companies have reason to worry about
hardware expenditures, but facebook's is more imminent

------
jeffesp
I find the introduction of a hybrid polling/interrupt driven network interface
really interesting. Mainly because we did the same work in 2003 on NetBSD for
a wireless router that never made it to market. We were at the opposite end of
the spectrum. Basically, we were running on a 100 MHz 486 and spent too much
time on interrupts with all the network traffic. It's funny that it is the
same situation for them on the latest systems out there (of course for some of
the same, but some different reasons as well).

------
lallysingh
Also, for all that complaining about Linux, anyone look at other Unices?

------
retyred
used to work with paul saab (ps) at yahoo. super smart as this post shows.
could never get them to use freebsd paul? :)

------
wmf
_Because we have thousands and thousands of computers, each running a hundred
or more Apache processes, we end up with hundreds of thousands of TCP
connections open to our memcached processes._

I guess using a single-process Web server would be too easy; it's practically
cheating.

~~~
look_lookatme
Wow, you're a genius. Facebook should hire you.

