Hacker News new | past | comments | ask | show | jobs | submit login
The WhatsApp Architecture Facebook Bought For $19 Billion (highscalability.com)
276 points by bussetta on Feb 26, 2014 | hide | past | web | favorite | 117 comments

I know this whole "$19B for whatsapp" topic is getting old already but I really think they solved some core problems that users had. Technically, those were probably not as challenging but functionally, this is what won me over :

- No registration required. Yes and Yes. do you hear Skype ? All you need is your phone number. And let's not get started on the privacy thing because the benefits outweigh the cost. See more below

- Easy to invite others. Just sends a simple SMS/text and wala, you are in.

- In countries like the US, SMS/Texting is very expensive (yes, i know...). Specifically, if you want to talk to someone international. Whatsapp solved this problem with ease. There were others (viber etc) but they just don't have the easy interface at least in my experience. Sure, the caveat is that everyone needs to have wifi/data plans but those are getting much more easier than convincing your phone company to stop charging bullshit rates for a simple text.

- Lets you create Groups. This sounds simple but imagine wanting to talk to your core family members in 1 text ? Whatsapp lets you do that with no pain.

- Share pictures/multimedia within groups. Again, very simple to use and comes included. I don't want to even explain what these cell phone companies charge for something they lable as MMS (not text).

Overall, when I came across Whatsapp, I was like "Finally, someone gets it and this should have been so duhhh but apparently, it wasn't". Very happy for the entire whatsapp team.

Sorry, I'm normally not one of those people who correct others on spelling/grammar, but this is a pet peeve of mine: it's "voila" (french), not "wala".

It almost makes sense if he meant "wallah"

"""Just sends a simple SMS/text and I promise by God, you are in."""


He didn't mean that of course...

Another possible misunderstanding is that "wala" in Filipino means to be missing/absent.

I was confused for a moment before I thought of "voila".

hahahahahaha. I love this thread.

It's actually "voilà" in real French... :P

ah , mais tout le monde peut se tromper,surtout avec le français, ou il est difficile de savoir comment un mot s'écrit tant qu'on ne l'a pas lu...

où ;)

Your french is pretty good!

L'arroseur devient l'arrosé ... ahah

Yep. And it's "per se" (Latin for: "in itself", "as it is") and not "per say" which doesn't mean anything at all.

Usually with loan words like this that are used enough, a spelling transformation is inevitable. I mean, I am sure voila itself is someone's pet peeve since its really "vois là". :)

I'd missed that initially; I'm amused that someone's gone all this time thinking "wala" is just a noise people make to indicate "done!"

Oh, that's what he meant.

Unless it's a wordplay, like Wuala(.com).

Is that what that's supposed to be? I always pronounced it "Woo-allah" in my head.

I'm not sure if that's what it's supposed to be. But that's what I made of it. :)

I was going to say the same... except it's "voilà"

Muphry's law strikes again!

> Lets you create Groups. This sounds simple but imagine wanting to talk to your core family members in 1 text ? Whatsapp lets you do that with no pain.

That is the #1 feature for my family. I'm in the US, my sister is in India, and our parents travel all over the world. Nearly every day, we post something to our group chat. Be it photos, personal news updates, or questions to one another, we actively use it almost daily. I have an iPhone and I think they have Androids.


Siblings in England, USA and Australia and we all use the group chat.

"Easy to invite others. Just sends a simple SMS/text"

and then

"In countries like the US, SMS/Texting is very expensive...Whatsapp solved this problem"

You need texting (a X dollar/month service) to use the service ... which solves the problem of... expensive texting?

I always found that to be somewhat odd, using a service you're trying to replace/compete with as the sole authenticating factor.

Edit: I realized after actually using the app, they do in fact offer an alternative authentication method (call). You have to try (and fail) at the texting method first.

Also, in response to derefr (I deleted my ignorant parent comment), I block texting as even if you don't pay for a plan, you still get random spam texts that you must page for (even if you didn't ask for them)

You send a text to invite someone to download the app and then you transition all your discussion over to their network instead of the SMS network. Hence solving the problem of expensive texts.


I've never seen a phone provider that doesn't still let you receive texts when you aren't paying for a text plan. They're just stupidly expensive to send/receive without a plan. (E.g. $0.50 per SMS.)

Paying for received texts is a US-only thing.

And the crappiest of all, if you're on a smartphone, you get charged for those texts even if you don't open them.

Pretty sure you get charged for texts on dumb phones too

>You need texting (a X dollar/month service) to use the service ... which solves the problem of... expensive texting?

Yes. Because texting in most countries has limits (e.g 200 sms per month, for more you pay an amount per text) while you can use WhatsApp for as many texts as you like.

And texting includes MMS (texting with images, sound etc), which is free in WhatsApp, but costs extra in most countries (can even go like $1 per text or more).

What I mean is that most of us already have data plans these days and we are already paying for it. Why pay extra for texting when we can use something like whatsapp that uses the data plan. Not to mention the crazy International texting charges. Even if you have unlimited texting from say T-mobile, you will pay for international texts because the unlimited is only for domestic texts.

It's somewhat of a weaker argument, considering that in most of the world MTs (mobile-terminated messages) are free, with charges applying only for MOs (mobile-originated).

Also, even in the US, a combo voice+text plan with heavy texting is still going to be cheaper for a combo voice+data plan, so average user ends up paying more money, not less.

People already have a data plan for other reasons.

Mobile plans vary significantly between countries. Using the US pricing structure as an example simply doesn't work. Just because a data plan is more expensive in the US doesn't necessarily mean it will be more expensive elsewhere.

Yes. What I meant was: voice + sms plan does not replace a data plan. A data plan plus WhatsApp (and perhaps Skype) can replace a voice + sms plan.

Big difference between a one-time invite cost and the cost of continually texting/messaging someone.

I have a phone with no data plan so I can't use WhatsApp just to let you know... :D

In countries like Indonesia where SMS is almost free, they still use Whatsapp because of the Network effect.

Groups is the biggest one. It helps SELLING. YES, call it e-Commerce 3.0! This is how HOUSEWIVES sell stuffs. Through Whatsapp OR BBM.

>Groups is the biggest one. It helps SELLING. YES, call it e-Commerce 3.0! This is how HOUSEWIVES sell stuffs. Through Whatsapp OR BBM.

Can you please elaborate on this.. How does it work? Any links?

What needs to elaborate? You 'spam' the groups with "new items, coming next week, here's the pic and price"

Something similar to this: http://storify.com/cbccommunity/kuwait-entrepreneurs-use-ins...

In the old days, you sent SMSes but you can't spam multiple people at once. You can only send one SMS at a time (unless you program something, which Housewives don't do...)

BBM supports Groups already but it's kind of slow (at least the old BBM anyway). I don't know if WhatsApp is any faster or whether WhatsApp has any limit on how many Groups member you can have (because it is essentially almost like Twitter).

There's a 50 user limit in a WhatsApp group.

different poster (singapore)

50 is a good number of people, especially if you're doing sample sales i.e. 1pm - 4pm special sales in a secret location by XX designer. For instance you can whatsapp a group of well-known fashion bloggers, who may forward the invite to a group of friends etc etc creating a semi-exclusive shopping experience

I've seen people do it for yard sales too (what we call void deck sales) It's a bit like a tupperware party.

And you forgot about the capability of sending other rich content like a map! ;-)

how long until mobile phones are purely data devices and cannot make calls or send texts, but do all of that via apps?

isn't that why facebook wanted WhatsApp, as they see this is a future of mobile phones?

"Erlang rocks! Erlang continues to prove its capability as a versatile, reliable, high-performance platform. Though personally all the tuning and patching that was required casts some doubt on this claim."

Really? Because I would trust their claims all the more; they've been in the trenches, they had expertise in other languages (from the article, "Having built a high performance messaging bus in C++ while at Yahoo, Rick Reed is not new to the world of high scalability architectures."), and then they became -intimate- with the language, in removing bottlenecks to their specific use case...and they still make the claim Erlang is great in this domain. Whose opinion are you going to credit more than theirs?

(Also, I bet many of the patches they made that weren't specific to their use cases were submitted as patches to the BEAM)

What it means is that unless you are comfortable modifying Erlang itself (BEAM, internal data structures, hash table allocations, etc.), then your install of Erlang won't scale like WhatsApp. You'll also need a ton of monitoring because Erlang definitely isn't of the "just turn it on, it'll be fine" variety. Nothing at that scale is.

That's fine, but people do tend to oversell Erlang IMO. The types of things Rick Reed talks about in his talk are standard issue stuff. Lock contention? Okay, how about using an architecture that doesn't require locks in the first place?

With Erlang, you're forced into the Erlang way from day one, and it'll be tough to escape. The good news is that Erlang has a good design overall, so as long as you can modify the engine under the hood, you'll be fine.

I don't think anyone should be scared away from Erlang, just be aware what you're getting into—like any other tech you could deploy.

Hard to avoid lock contention when it's happening in the OS.

Yup, that's why you skip that too:


Yap some patches or suggestions made it in and the greater community benefited. Some lock contention points removed, memory management stuff etc.

Also remember this is just one visible project. Erlang VM is being developed actively and many other companies and people are contributing. For an almost 30 year old language it very active. Maps are coming in R17 so are dirty schedulers, {active,N} on sockets, lots of goodies.

I know; R17-rc1 was tagged a month ago.

This was more my being agog at the author viewing the work that was done to the BEAM as evidence -against- Erlang being great, rather than the fact that these people with that level of experience stayed with it, and still raved about it, as evidence in Erlang's favor. And that the issues they encountered are probably fixed any way, or apply only to their use case (and so may still exist, but have a tradeoff that is worth keeping in other use cases).

Omg dirty schedulers are included too? I need a new pair of pants :x

Dealing with NIF threads just feels dirty

if someone is looking for more info on dirty-scheduler, here is slightly oldish slide: http://www.erlang-factory.com/upload/presentations/377/Ricka...

The whole error handling philosophy of Erlang is extremely fascinating. The fact that it was born out of the telecommunication world makes it extremely fit for this kind of job. I've been reading "Programming Erlang" on and off for a few weeks now, and it's... eye opening. Erlang proves functional programming is for the real, real world. There isn't a language that's more down to earth, ready to "Get Shit Done" than Erlang.

Edit: When you learn Erlang you'll notice every design decision was made for performance or reliability (vs making the language "nice", think Ruby). I appreciate that.

I think the golden moment for me thinking about Erlang was when I started thinking that the actor model was made to support fault tolerance and concurrency was a nice bonus.

Indeed, for all the free wins you get from Erlang's actor model (e.g. easy just-add-cores parallelism), you also get losses (e.g. copy-based message-passing is pretty much the least performant signalling mechanism possible.) But when you look at it all through the lens of fault-tolerance, there's really nothing you can change about Erlang's semantics without throwing away some guarantee or another.

When I hear that other languages (e.g. Go) have "actors like Erlang", I have to laugh. Unless you're willing to eat all the disadvantages and performance losses that come with Erlang's approach, you don't really have "actors like Erlang"; you just have green threads.

It's downright adorable when people compare a half-baked language like Go to Erlang/OTP

Yeah Joe makes that point pretty often, and it was a nice revelation for me as well.

Actor concurrency paradigm has been copies by other languages and libraries. Not many managed to copy the fault tolerance and transparent distribution aspect. Granted there are other ways to solve that (OS processes, VMs) but still, it is interesting to learn from and think about.

Something drôle I caught on Joe Armstrong's twitter feed: ``A machine learning researcher, a crypto-currency expert, and an Erlang programmer walk into a bar. Facebook buys the bar for $27 billion.''

What are the other acquisitions/hires referenced? :-)

But aren't most of those messages point-to-point? Not that I'm trivializing 70M/sec! But for the most part, it's pretty straightforward right? No multi-client support (so no desktop client that requires server sync). Most messages(?) are to one recipient only, right? So send the message, send an ACK, deliver message, deliver read ACK? It seems like the perfect thing for a scale-out architecture. Even group chat can probably just be split into multiple messages, one per recipient.

Not that it's not a lot of work, but isn't most of it just hard work engineering? Add hardware, deal with bottlenecks. Versus having to come up with some novel new datastructures and inter-server communications?

SMTP sends a ton of messages, too. I don't see why an IM system couldn't take essentially the same concept, where one small cluster of services is responsible for the mailboxes for certain individuals.

Phone signaling is also just making calls point to point. Very simple. Well except that making it not crash, making it scale, handling corner cases, legacy devices, regulatory stuff.

One can sort of take a high level look at a large class of system and just say "it sends messages from one part to another". A financial transaction sends one message from one account to another one. A lot of sites and businesses are "just" get data from database send to web page, then send updates back to the database. Pretty simple.

The devil is in the details and in the proof of work. If it is simple, ok, do it. Not being facetious, you might actually succeed. Just try it. Erlang is a good choice for it, start with that.

> SMTP sends a ton of messages, too. I don't see why an IM system couldn't take essentially the same concept,

Hint's in the name: instant messaging. For the most part, nobody stops using SMTP if it takes 2 minutes to deliver an e-mail

>Hint's in the name: instant messaging. For the most part, nobody stops using SMTP if it takes 2 minutes to deliver an e-mail

Well, I don't think anybody would stop using IM too, if it took say 10 or 20 seconds to deliver an IM.

Honestly, you might not immediately, but if it happened often enough, you probably would switch clients. Back in the day, if AIM managed to screw up enough of mine and my friend's conversations by not sending messages quickly enough (and then sending 30 all at the same time), we'd have moved off it if something better was found.

I think this was a big factor in why AIM trumped ICQ in America. ICQ had that "sending mail" icon that took a few seconds (or more) to "send" your message and it didn't quite feel instant.

It was also a ui issue. ICQ originally had an interface which didn't show a running log of the conversation. It was more like sending email with presence indicators and a 450 character limit. Though it also had an IRC-like mode that was rarely used. I remember finding AIM and MSN so different when they emerged as the 'instant' aspect, typing indicator, and running log encouraged rapid-fire short messages and gave a conversation-like feel.

> ICQ originally had an interface which didn't show a running log of the conversation.

The earliest version I remember, before 98a, had a history button that would show the running log on the top of the message box window, which was nice.

The nicest thing about ICQ for me back then was the fact that the message disappeared when you sent it -- it wasn't there filling your taskbar or occupying your mindspace.

Good! you got the technology, now go get those 450 million users :)

> "Ejabberd is a famous open source Jabber server written in Erlang… The next few years were spent re-writing and modifying quite a few parts of ejabberd, including switching from XMPP"

Anyone looked at what WhatsApp sends over the wire these days? I wonder what it is? A binary protocol?

XMPP with a homegrown compression layer.

It's so awesome to see FreeBSD as the OS. Hard to learn in my opinion but worth it since it's so reliable.

My experience was it was a lot easier to learn than various linux distros, at least several years ago. It has great documentation in the form of the Handbook, and its directory structure & the ports system are rational & predictable.


In terms of learning it's more stable than various Linux distros.

So in a way, it's easier to learn FreeBSD than Linux (Server) distros.

Does anyone know what service WhatsApp used for SMS delivery for phone number verification? I wish I could get text messages into all parts of the world like they do.

At Bolt (https://bolt.co) we use Twilio for this, and I'm pretty sure WhatsApp does too along with other providers. With Twilio we also fall back to delivering access tokens with a phone call when the SMS fails.

(Full disclosure: I am a former Twilio employee.)

Fuller disclosure: you're just another spammer.

Authy (a two-factor auth service) uses a combination of Twilio and BulkSMS (they say that BulkSMS is particularly good for Latin America). Authy explains how they handle global SMS here: http://blog.leanstack.io/how-authy-built-a-fault-tolerant-tw...

They used multiple providers.

http://nexmo.com has pretty decent coverage

Not really, there wasn't even a category for France and several first world countries did not have any available phone numbers to buy.

I work for a company called mblox; we put all our effort into reach.

SNS by AWS does it too.

"And using your cell number as identity and your contacts list as a social graph is diabolically simple."

Tthe linkage of phone number and identity seems to be the main reason why there are no tablet versions of the whatsapp client. I wonder how satisfied whatsapp/facebook are with this constraint, and whether they intent to address it? It is possible to imagine workarounds for devices that don't have a phone number, but they don't seem as friction-free as the current identity system.

You should have a look at how Line does it. You can use a login and password, but you can also use Line on another device through some QR code scanning. No password required IIRC.

I wonder how Facebook will deal with integrating Erlang into their (afaik) PHP-based systems?

facebook chat was originally written in erlang.


I doubt they'll have any trouble at all integrating.

I believe that the reason they switched away from Erlang, though, was that nearly nobody in the company (besides the team that made the prototype) knew the language. In other words, they switched away because they had trouble integrating Erlang into their processes, and instead chose to rewrite it using tools everyone else was more familiar with.

This, then, is basically recreating the same problem they had before, just with 40 or so more engineers who are on the "Erlang" side of the vote.

Yariv Sadan is already there, though. That plus the WhatsApp people makes for a pretty intimidating group of erlangers.

edit: Just found out Yariv left. Still, Facebook was using ejabberd, and WhatsApp built their business around ejabberd.

PHP is not the only language in use at Facebook right now...


They invented Thrift, I think they'll figure it out.

Of all Facebook's accomplishments why is Thrift the one that you think makes this a non-issue?

They literally built it to solve problems like this. From the project page, it was designed to provide "scalable cross-language services development".

Service oriented architecture, and avoiding or porting fat client libraries. Standard stuff.

they will do what comes naturally - take five years write a dual erlang php vm in D and then open source it randomly a year or so late.

The backend is a mix of things.

I read like 8 paragraphs of blather before I got to the first bit of useful technical information, a confirmation that they were using Erlang.

"Desktop is dead and web is dying" is very pessimistic.

From a global perspective, is it?

>By the end of this year, 6% of the global population will own a tablet, 20% will own PCs, and 22% will own smartphones.


Smartphones and tablets have browsers too, and they are by far the most used app on any platform.

Unless the phone already has a facebook and an amazon app, in which case the browser drops to 15th most used.

Interesting. Source?

Software is not a zero-sum game, so yes it's not just pessimistic. It's wrong.

Is it that time again?

if the web primarily became a data-layer for mostly native apps would that really be such a bad thing?

I'd say yes. We've been down that path before and i strongly believe that native apps are needed in a relatively low number of cases when you either need all the performance in your app or when you're dealing with some very tricky input. The rest should be absolutely fine with apps built with browser technologies and adapted to OS (with Apache Cordova or whatnot). Just as with WhatsApp - that app is pretty straightforward and it could be easily written in HTML5. It doesn't require heavy computation, it doesn't have any complex input methods, it doesn't need to deal with local storage etc.

Web allows us to create one app and with tiniest of modifications that app could be available on all of the major platforms. Don't you think this is by far better future than multitude of various platforms and versions of those platforms (see WP, Android) when you'll need a team of developers just to finish multiple versions of your app?

Yes, because overloading HTTP(S) as a data layer for native apps leaves a ton of performance and security on the table. It's for web browsers, not every possible use case for sending a buffer of bytes from A to B.

People use HTTP because it's what they know, not because it's technically a good (efficient, reliable, scalable) approach for their particular use case.

It does make sense for apps that also have a web component, but that's becoming less and less necessary. When you can drop it, the possibilities are pretty amazing having a fully-capable computer with CPU equivalent to a 2010 Macbook Air at your disposal at the edge of the network.

Hell, my latest project doesn't even use DNS.

This, a million times.

I had to fight more than once against management in order not to use HTTP for everything. HTTP isn't even that simple to deal with. Good thing is that the numbers were always with me, and with code examples, it was always doable, but it's incredible how HTTP got its way in people's minds!

Nope, computer end-points keep going back and forth between thick client and thin client. They both have advantages and disadvantages, depending on relative cost of hardware and how hard it is to maintain the code.

Man, I burn with envy when read such stories of monumental technical achievements. For some reason have always been fascinated by scalability. But so far life has given me the opportunity in a very limited way.

Have been reasonably proud of scaling my own micro (in comparison) service. But right now feel very small and humbled.

edit: minor

This article also mentions about whatsapp in business usages. Interestingly, all those points resonate with a messenger project that we are working on. http://peer.im

First 1/3 of the article is talking about background or $16B?!?... Much of the rest is a list of hardware. Be prepared to skim a ton to find the actual subject of the article.

As Facebook transitions into the next decade, they must focus heavily on mobile. I see a lot of people mainly using Facebook for its messenger app, so it's very smart of Facebook to buy a competing messaging service that is gaining huge traction in the international market. If WhatsApp were to expand into the States, Facebook could be in trouble. Very smart investment by Zuckerberg but I can see why people are shocked it was dealt for such a huge amount.

The performance levels they have achieved are impressive.

I think hot swapping code is a major factor in being able to quickly fix and develop a system like this.

I wonder how far one can go on another tech stack.

For example Akka is supposed to be able to handle up to 50m messages on a single box and there is some support for hot swapping in the Oracle and IBM runtime.

Does anyone know whether their changes to the BEAM ever made into the official Erlang BEAM?

Interesting enough to check into Erlang. Even though most of us won't approach the load Whatsapp has to handle, it is very useful to understand how to pack more work into smaller hw footprints for any size business.

I think they owe the developers of ejabberd a few drinks

WhatsApp = Email 2.0?

No Linux? Am I dreaming?

does anyone know where they were hosted? did they colo or use a msp?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact