

Behind the Scenes at Facebook: Scaling Up FBChat Using Erlang - Anon84
http://education.sys-con.com/node/634250

======
jfarmer
FWIW, this is just a republication of a 3-month-old blog post:
[http://www.facebook.com/note.php?note_id=14218138919&id=...](http://www.facebook.com/note.php?note_id=14218138919&id=9445547199&index=2)

------
KirinDave
I'm surprised to hear that they're using the alt erlang-thrift binding in
production. In Powerset's application of it, it begins to rattle and break
down when you push it about 90r/s. I wonder if FBChat's per-connection load is
lower than that?

Thrift is a great idea and I've used it alot, but it's frustrating to see it
so tied to C++. The idea of code generation for your IDL interfaces is so last
decade. We've got experimental thrift bindings that treat the IDL as a
runtime-compiled source, so on the Ruby or Erlang side you can use their
dynamic strong typing to avoid the necessity of code generation that the C++
side suffers.

~~~
ComputerGuru
Current fbChat loads are really bad.

I believe there's some sort of prioritization going on - connecting from a
"3rd world" ISP to Facebook gives me some really bad lag and disconnects,
while going through a VPN (server in LA) provides much better response times.
If it were just this one ISP, I'd blame it on them; but in my months of using
fbChat, I've noticed some clear scaling problems for their "real-time" service
compared to the other, more static parts of their site.

Obviously this isn't a scientific study in any way, but fbChat has always been
a bad performer.

~~~
neilc
Does anyone actually _use_ Facebook chat? Leaving aside the details of the
implementation, chatting via the FB web interface seems terribly awkward. And
while _I_ can use XMPP, most of the people you'd be chatting with won't be.

------
stcredzero
Wow. Testing with a "Dark Launch." Basically, they used cycles from every user
machine with an open Facebook page to do their load testing by distributing a
version of their app with no UI and a test script. This is very powerful and
kind of scary.

~~~
arockwell
Google did something really similar with their gchat launch on Orkut. They
started out with just 1% of the userbase running it in the background and
scaled it up from there till they were confident that the implementation
details were mostly worked out. Its really a great idea for testing the
scalability of new features without completely breaking the production site.

------
axod
I wonder what their stats are like so far. Are people using it to chat?

