
The WhatsApp Architecture Facebook Bought For $19 Billion - bussetta
http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html
======
codegeek
I know this whole "$19B for whatsapp" topic is getting old already but I
really think they solved some _core_ problems that users had. Technically,
those were probably not as challenging but functionally, this is what won me
over :

\- No registration required. Yes and Yes. do you hear Skype ? All you need is
your phone number. And let's not get started on the privacy thing because the
benefits outweigh the cost. See more below

\- Easy to invite others. Just sends a simple SMS/text and wala, you are in.

\- In countries like the US, SMS/Texting is very expensive (yes, i know...).
Specifically, if you want to talk to someone international. Whatsapp solved
this problem with ease. There were others (viber etc) but they just don't have
the easy interface at least in my experience. Sure, the caveat is that
everyone needs to have wifi/data plans but those are getting much more easier
than convincing your phone company to stop charging bullshit rates for a
simple text.

\- Lets you create Groups. This sounds simple but imagine wanting to talk to
your _core_ family members in 1 text ? Whatsapp lets you do that with no pain.

\- Share pictures/multimedia within groups. Again, very simple to use and
comes included. I don't want to even explain what these cell phone companies
charge for something they lable as MMS (not text).

Overall, when I came across Whatsapp, I was like "Finally, someone gets it and
this should have been so duhhh but apparently, it wasn't". Very happy for the
entire whatsapp team.

~~~
berberous
Sorry, I'm normally not one of those people who correct others on
spelling/grammar, but this is a pet peeve of mine: it's "voila" (french), not
"wala".

~~~
gahahaha
It almost makes sense if he meant "wallah"

"""Just sends a simple SMS/text and I promise by God, you are in."""

[http://en.wikipedia.org/wiki/Wallah_(Arabic)](http://en.wikipedia.org/wiki/Wallah_\(Arabic\))

He didn't mean that of course...

~~~
dorian-graph
Another possible misunderstanding is that "wala" in Filipino means to be
missing/absent.

I was confused for a moment before I thought of "voila".

------
lostcolony
"Erlang rocks! Erlang continues to prove its capability as a versatile,
reliable, high-performance platform. Though personally all the tuning and
patching that was required casts some doubt on this claim."

Really? Because I would trust their claims all the more; they've been in the
trenches, they had expertise in other languages (from the article, "Having
built a high performance messaging bus in C++ while at Yahoo, Rick Reed is not
new to the world of high scalability architectures."), and then they became
-intimate- with the language, in removing bottlenecks to their specific use
case...and they still make the claim Erlang is great in this domain. Whose
opinion are you going to credit more than theirs?

(Also, I bet many of the patches they made that weren't specific to their use
cases were submitted as patches to the BEAM)

~~~
rdtsc
Yap some patches or suggestions made it in and the greater community
benefited. Some lock contention points removed, memory management stuff etc.

Also remember this is just one visible project. Erlang VM is being developed
actively and many other companies and people are contributing. For an almost
30 year old language it very active. Maps are coming in R17 so are dirty
schedulers, {active,N} on sockets, lots of goodies.

~~~
lostcolony
I know; R17-rc1 was tagged a month ago.

This was more my being agog at the author viewing the work that was done to
the BEAM as evidence -against- Erlang being great, rather than the fact that
these people with that level of experience stayed with it, and still raved
about it, as evidence in Erlang's favor. And that the issues they encountered
are probably fixed any way, or apply only to their use case (and so may still
exist, but have a tradeoff that is worth keeping in other use cases).

------
AlexanderDhoore
The whole error handling philosophy of Erlang is extremely fascinating. The
fact that it was born out of the telecommunication world makes it extremely
fit for this kind of job. I've been reading "Programming Erlang" on and off
for a few weeks now, and it's... eye opening. Erlang proves functional
programming is for the real, real world. There isn't a language that's more
down to earth, ready to "Get Shit Done" than Erlang.

Edit: When you learn Erlang you'll notice every design decision was made for
performance or reliability (vs making the language "nice", think Ruby). I
appreciate that.

~~~
Glide
I think the golden moment for me thinking about Erlang was when I started
thinking that the actor model was made to support fault tolerance and
concurrency was a nice bonus.

~~~
derefr
Indeed, for all the free wins you get from Erlang's actor model (e.g. easy
just-add-cores parallelism), you also get losses (e.g. copy-based message-
passing is pretty much the least performant signalling mechanism possible.)
But when you look at it all through the lens of fault-tolerance, there's
really nothing you can change about Erlang's semantics without throwing away
some guarantee or another.

When I hear that other languages (e.g. Go) have "actors like Erlang", I have
to laugh. Unless you're willing to eat all the disadvantages and performance
losses that come with Erlang's approach, you don't really have "actors like
Erlang"; you just have green threads.

~~~
wcummings
It's downright _adorable_ when people compare a half-baked language like Go to
Erlang/OTP

------
_nato_
Something drôle I caught on Joe Armstrong's twitter feed: ``A machine learning
researcher, a crypto-currency expert, and an Erlang programmer walk into a
bar. Facebook buys the bar for $27 billion.''

~~~
scott_karana
What are the other acquisitions/hires referenced? :-)

------
MichaelGG
But aren't most of those messages point-to-point? Not that I'm trivializing
70M/sec! But for the most part, it's pretty straightforward right? No multi-
client support (so no desktop client that requires server sync). Most
messages(?) are to one recipient only, right? So send the message, send an
ACK, deliver message, deliver read ACK? It seems like the perfect thing for a
scale-out architecture. Even group chat can probably just be split into
multiple messages, one per recipient.

Not that it's not a lot of work, but isn't most of it just hard work
engineering? Add hardware, deal with bottlenecks. Versus having to come up
with some novel new datastructures and inter-server communications?

SMTP sends a ton of messages, too. I don't see why an IM system couldn't take
essentially the same concept, where one small cluster of services is
responsible for the mailboxes for certain individuals.

~~~
nly
> SMTP sends a ton of messages, too. I don't see why an IM system couldn't
> take essentially the same concept,

Hint's in the name: _instant_ messaging. For the most part, nobody stops using
SMTP if it takes 2 minutes to deliver an e-mail

~~~
coldtea
> _Hint 's in the name: instant messaging. For the most part, nobody stops
> using SMTP if it takes 2 minutes to deliver an e-mail_

Well, I don't think anybody would stop using IM too, if it took say 10 or 20
seconds to deliver an IM.

~~~
mjmahone17
Honestly, you might not immediately, but if it happened often enough, you
probably would switch clients. Back in the day, if AIM managed to screw up
enough of mine and my friend's conversations by not sending messages quickly
enough (and then sending 30 all at the same time), we'd have moved off it if
something better was found.

~~~
atgm
I think this was a big factor in why AIM trumped ICQ in America. ICQ had that
"sending mail" icon that took a few seconds (or more) to "send" your message
and it didn't quite feel instant.

~~~
krrrh
It was also a ui issue. ICQ originally had an interface which didn't show a
running log of the conversation. It was more like sending email with presence
indicators and a 450 character limit. Though it also had an IRC-like mode that
was rarely used. I remember finding AIM and MSN so different when they emerged
as the 'instant' aspect, typing indicator, and running log encouraged rapid-
fire short messages and gave a conversation-like feel.

~~~
atgm
> ICQ originally had an interface which didn't show a running log of the
> conversation.

The earliest version I remember, before 98a, had a history button that would
show the running log on the top of the message box window, which was nice.

The nicest thing about ICQ for me back then was the fact that the message
disappeared when you sent it -- it wasn't there filling your taskbar or
occupying your mindspace.

------
sjtgraham
> "Ejabberd is a famous open source Jabber server written in Erlang… The next
> few years were spent re-writing and modifying quite a few parts of ejabberd,
> including switching from XMPP"

Anyone looked at what WhatsApp sends over the wire these days? I wonder what
it is? A binary protocol?

~~~
Zash
XMPP with a homegrown compression layer.

------
snake_plissken
It's so awesome to see FreeBSD as the OS. Hard to learn in my opinion but
worth it since it's so reliable.

~~~
jambo
My experience was it was a lot easier to learn than various linux distros, at
least several years ago. It has great documentation in the form of the
Handbook, and its directory structure & the ports system are rational &
predictable.

[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/](http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/)

------
tlack
Does anyone know what service WhatsApp used for SMS delivery for phone number
verification? I wish I could get text messages into all parts of the world
like they do.

~~~
stanleydrew
At Bolt ([https://bolt.co](https://bolt.co)) we use Twilio for this, and I'm
pretty sure WhatsApp does too along with other providers. With Twilio we also
fall back to delivering access tokens with a phone call when the SMS fails.

(Full disclosure: I am a former Twilio employee.)

~~~
username223
Fuller disclosure: you're just another spammer.

------
andyjohnson0
_" And using your cell number as identity and your contacts list as a social
graph is diabolically simple."_

Tthe linkage of phone number and identity seems to be the main reason why
there are no tablet versions of the whatsapp client. I wonder how satisfied
whatsapp/facebook are with this constraint, and whether they intent to address
it? It is possible to imagine workarounds for devices that don't have a phone
number, but they don't seem as friction-free as the current identity system.

~~~
uniclaude
You should have a look at how Line does it. You _can_ use a login and
password, but you can also use Line on another device through some QR code
scanning. No password required IIRC.

------
ForHackernews
I wonder how Facebook will deal with integrating Erlang into their (afaik)
PHP-based systems?

~~~
chollida1
facebook chat was originally written in erlang.

[http://www.erlang-
factory.com/upload/presentations/31/Eugene...](http://www.erlang-
factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf)

I doubt they'll have any trouble at all integrating.

~~~
derefr
I believe that the reason they switched away from Erlang, though, was that
nearly nobody in the company (besides the team that made the prototype) knew
the language. In other words, they switched away _because_ they had trouble
integrating Erlang into their processes, and instead chose to rewrite it using
tools everyone else was more familiar with.

This, then, is basically recreating the same problem they had before, just
with 40 or so more engineers who are on the "Erlang" side of the vote.

~~~
pessimizer
Yariv Sadan is already there, though. That plus the WhatsApp people makes for
a pretty intimidating group of erlangers.

edit: Just found out Yariv left. Still, Facebook was using ejabberd, and
WhatsApp built their business around ejabberd.

------
austinz
I read like 8 paragraphs of blather before I got to the first bit of useful
technical information, a confirmation that they were using Erlang.

------
nikentic
"Desktop is dead and web is dying" is very pessimistic.

~~~
jere
From a global perspective, is it?

>By the end of this year, 6% of the global population will own a tablet, 20%
will own PCs, and 22% will own smartphones.

[http://www.businessinsider.com/smartphone-and-tablet-
penetra...](http://www.businessinsider.com/smartphone-and-tablet-
penetration-2013-10)

~~~
grey-area
Smartphones and tablets have browsers too, and they are by far the most used
app on any platform.

~~~
pessimizer
Unless the phone already has a facebook and an amazon app, in which case the
browser drops to 15th most used.

~~~
oakesm9
Interesting. Source?

------
rehack
Man, I burn with envy when read such stories of monumental technical
achievements. For some reason have always been fascinated by scalability. But
so far life has given me the opportunity in a very limited way.

Have been reasonably proud of scaling my own micro (in comparison) service.
But right now feel very small and humbled.

edit: minor

------
angelohuang
This article also mentions about whatsapp in business usages. Interestingly,
all those points resonate with a messenger project that we are working on.
[http://peer.im](http://peer.im)

------
grogenaut
First 1/3 of the article is talking about background or $16B?!?... Much of the
rest is a list of hardware. Be prepared to skim a ton to find the actual
subject of the article.

------
tuf14575
As Facebook transitions into the next decade, they must focus heavily on
mobile. I see a lot of people mainly using Facebook for its messenger app, so
it's very smart of Facebook to buy a competing messaging service that is
gaining huge traction in the international market. If WhatsApp were to expand
into the States, Facebook could be in trouble. Very smart investment by
Zuckerberg but I can see why people are shocked it was dealt for such a huge
amount.

------
izarov
The performance levels they have achieved are impressive.

I think hot swapping code is a major factor in being able to quickly fix and
develop a system like this.

I wonder how far one can go on another tech stack.

For example Akka is supposed to be able to handle up to 50m messages on a
single box and there is some support for hot swapping in the Oracle and IBM
runtime.

------
plainOldText
Does anyone know whether their changes to the BEAM ever made into the official
Erlang BEAM?

------
dpweb
Interesting enough to check into Erlang. Even though most of us won't approach
the load Whatsapp has to handle, it is very useful to understand how to pack
more work into smaller hw footprints for any size business.

------
jamespo
I think they owe the developers of ejabberd a few drinks

------
retube
WhatsApp = Email 2.0?

------
gwu78
No Linux? Am I dreaming?

------
beachstartup
does anyone know where they were hosted? did they colo or use a msp?

~~~
yonasb
They're hosted on SoftLayer [http://leanstack.io/cloud-
stacks/whatsapp](http://leanstack.io/cloud-stacks/whatsapp). Source:
[http://www.ip-adress.com/whois/whatsapp.com](http://www.ip-
adress.com/whois/whatsapp.com)

