
Why WhatsApp Only Needs 50 Engineers for Its 900M Users - ghosh
http://www.wired.com/2015/09/whatsapp-serves-900-million-users-50-engineers/
======
chetanahuja
Basically they're a silicon valley company who've done everything that's
exactly the opposite of standard silicon valley advise -- bootstrapped for
first few years, complete apathy or even aversion to silicon valley marketing
(if you live in the valley and have no foreign connection, you probably didn't
even know what whatsapp was before 2012 or so), built for all client platforms
simultaneously (including blackberry and Nokia), focus on non-US markets
first, built out on leased hardware instead of public cloud, a simple pay-for-
use business model selling to _consumers_ (before facebook bought them), deep
aversion to ads ([http://blog.whatsapp.com/245/Why-we-dont-sell-
ads?](http://blog.whatsapp.com/245/Why-we-dont-sell-ads?)) etc. The fact that
their technology stack ended up being Erlang on FreeBSD is probably the least
distinguishing feature about them.

~~~
spyder
"built for all client platforms simultaneously"

And that's why it's strange that the article focuses on Erlang: Are all of the
client apps written in Erlang?

~~~
chetanahuja
Not at all. The client apps were written in their respective
framework/languages.

Yes yes, I do believe there's some boosterism going on when their success is
entirely attributed to Erlang/FreeBSD alone. Sure the stability and
scalability of the _backend_ may have been helped by using Erlang but the fact
that they got to scale at all has to do with all these other
business/cultural/timing/design factors that I mentioned above.

Btw, the one factor I forgot to mention above was the call to use phone
numbers as the namespace for user identity. I believe they were one of the
early ones, if not the first ones, who eschewed email and/or social login and
went straight to phone number. That made them a direct replacement for SMS in
large parts of the world where SMS costs were meaningfully expensive to the
population. One more example of focusing outside the US market in general and
outside of Valley bubble in particular.

------
sidcool
Comments in this thread are quite cringeworthy. People calling WhatsApp a
simple application seem to fail to understand the cost of simplicity under
such scale.

~~~
mpdehaan2
It wasn't a great article from wired. I feel the language choice is probably
the least contributing factor in terms of what's interesting at that scale.
Yes, it's an eclectic choice, but I find systems/component architecture much
more interesting than language and program architecture in scaling
considerations.

A bit more info here: [http://highscalability.com/blog/2014/3/31/how-whatsapp-
grew-...](http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-
nearly-500-million-users-11000-cores-an.html)

With regard to fun challenges to solve, from that article:

"Lots of problems related to scale as you might imagine. Problems with
flapping connections, queues getting so long they delay high priority
operations, flapping of timers, code that worked just fine at one traffic
level breaking badly at higher traffic levels, high priority messages not
getting serviced under high load, operations blocking other operations in
unexpected ways, failures causing resources issues, and so on. These things
just happen and have to be worked through no matter what system you are
using."

~~~
rdtsc
I don't know I found the article decent _for_ being on wired. Didn't expect
BEAM VM internals there...

> I feel the language choice is probably the least contributing factor in
> terms of what's interesting at that scale.

However, as they say "right from the horses' mouths", WhatsApp engineers have
said that Erlang and FreeBSD are critical tools for their success. One can
accuse them of lying, of course, but why?

Here is an interview with Eugene Fooksman:

[https://pdincau.wordpress.com/2013/03/27/an-interview-
with-e...](https://pdincau.wordpress.com/2013/03/27/an-interview-with-e..).

\---

...and the consensus in our team is that it is largely because of Erlang.
We’re managing to serve huge amount of connections from single front-end
server...

\---

There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory
(already posted somewhere) in here.

Here is another older directly from their website:

[https://blog.whatsapp.com/196/1-million-is-
so-2011](https://blog.whatsapp.com/196/1-million-is-so-2011)

Clearly states Erlang and FreeBSD are important.

> These things just happen and have to be worked through no matter what system
> you are using.

Yeah but that is a bit too generalizing. It is like saying "well the language
is Turing Complete" argument, yet it is but one tool is much better than
another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you
can do it many other ways. And, at least in this case, they certainly picked
the right tools, as we can see from their success. The number of users *
amount of messages processed / # of engineers is a not a bad metric to see how
good a tool is. And Erlang is a great tool here.

(Another example is when you use the smartphone to access the Internet,
chances are 50% of the time that connection is set up by Erlang. It is there
in the background doing its job, not crashing, setting up even an order of
magnitude larger amount of data than we see from WhatsApp).

~~~
mpdehaan2
"One can accuse them of lying, of course, but why?"

I never said that.

------
ghshephard
If this isn't a counter to the inane recruiting requirements of "5 years with
language X" I don't know what is:

 _We don’t bring them in specifically because the engineer knows Erlang,”
Mahdavi said on Monday. “We expect the engineer to come in and spend their
first week getting familiar with the language and learning to use the
environment. If you hire smart people, they’ll be able to do that.”_

~~~
interpol_p
This reminds me of how we have been hiring engineers.

We sit them down for a day with a fairly simple task (build a simple web or
mobile app, about three screens with a basic data model) but we make sure to
give them a language or framework they are unfamiliar with.

They have Google, Stack Overflow, and other employees to use as a resource.
It's a great way to get to know new hires while seeing how resourceful and
interested in learning they are.

Every candidate has built the app to spec with a few hours remaining, but it's
really good to see those who take the initiative to spend a bit of extra time
polishing it and explaining it to us, and responding to our feedback.

~~~
mdellabitta
Does this interviewing methodology mean that you only hire devs that don't
currently have a gig or have vacation days to blow on the possibility of
getting a job with you?

~~~
interpol_p
We do have an initial hour meeting with candidates just to get to know them
and propose the trial if they seem like a possibility.

Wouldn't most people take a day off to interview somewhere? I know I wouldn't
be comfortable ducking out for a few hours to interview — it would feel tense
and uncomfortable.

Facebook, and I'm sure many other places, do full-day interviews. It's
certainly not unusual.

~~~
6d0debc071
> Wouldn't most people take a day off to interview somewhere?

Not always possible. One of the big annoyances when I was working for a
company was that if I wanted time off it was two weeks notice. Doesn't work
particularly well when the interview invitation tends to be more along the
lines of, 'Hi, your interview date is next week.'

~~~
humanrebar
Or maybe you have had many life events (births, deaths, etc.) and your
employer wasn't very understanding. So you've already burned all your time off
and are ready to find an employer who's a better fit.

------
chetanahuja
I'm surprised that nobody mentioned why whatsapp started with Erlang in the
first place. They started out with an open source xmpp package. The most
popular one at the time happened to be written in Erlang (ejabberd). There was
no masterplan to scale to close to a billion accounts.

As it happens, Erlang is an excellent platform upon which to build a message
exchange service. Also, the article doesn't mention their completely non-
traditional approach to ops. Despite the conventional wisdom in the valley,
they run their own servers. And squeeze an enormous amount of value from a
small number of servers. Their unique ops strategy is probably as much a part
of their success as simply using Erlang.

[http://www.erlang-
factory.com/static/upload/media/1394350183...](http://www.erlang-
factory.com/static/upload/media/1394350183453526efsf2014whatsappscaling.pdf)

------
planetjones
I don't quite understand the correlation. If the article was about how
Whatsapp achieves scale and parallelism through functional languages, then
fair enough. But I'm not sure it's fair to say they need 50 engineers, in
part, because of the language choice ? Surely they need 50 engineers because
their product is fairly simple and there's no need to have more engineers than
features. Maybe the point it should have made is that Whatsapp values higher
quality over quantity ? That I could relate to.

~~~
mrweasel
Actually I believe that answer is in the article: “Our strategy around
recruiting is to find the best and brightest engineers. We don’t bring them in
specifically because the engineer knows Erlang,”

WhatsApp only needs 50 engineers because of who they hire. Even using Erlang,
I pretty sure they would need more than 50 engineers if they hired mainly
around average engineers.

~~~
rifung
How do you know if an engineer is average vs one of the best?

I haven't heard of anyone who has figured out how to measure an engineer's
abilities.

Some people test algorithms. Others test how often they've shipped things.
Most just look for years of experience. A few even test whether you know
certain facts.

~~~
segmondy
Interviews!

~~~
rifung
I haven't heard of a single company that has nailed down interviews. I was
going to mention interviews but figured that if you couldn't measure how much
an employee contributes, you'd have a much harder time measuring how much a
prospective employee could contribute given a much shorter time to test them.

------
smcl
Isn't it more because they're a simple IM application (albeit spread across a
number of platforms)? I know they're hugely successful and there are
considerations about scaling to that sort of size, but surely the fact that
their product is extremely simple has to be a huge factor?

------
DanielBMarkham
(Disclaimer: I teach this stuff) Good article. It's a keeper. Touches on a lot
of things I've been telling clients for years.

My takeaways:

\- Pure functions in little composable pieces. Screw the fact you can't find
programmers. With 1/20th the staff, it's not important

\- Zero to few meetings. Keep focused on one thing. No playing around with
WhizzBang Platform 7.0 because all the cool kids are using it. Mind your
knitting.

\- Keep It Simple, Stupid. Engineers are our own worst enemy. Solve the most
simple problem possible. Then add complexity a little bit at a time, only as a
last resort. For those of you saying "But messaging is solved", "Erlang was
written for messaging", and so forth, this is where you missed. Any complex
system (with a few notable exceptions) will look very simple when solved in
this manner. In fact, that's the entire point.

\- Running and deploying are the same thing. CI/CD. There is no Big Red Button
to push because we're always working on the airplane engines while the plane
is flying

Coming from a traditional OOA/D/P background, this mentality was extremely
foreign to me. Even though I was using and talking about TDD, it was still in
terms of composing and re-arranging object graphs. It wasn't until I got into
F# several years ago that the pieces started coming together. This is a very
difficult conceptual leap to make, because a lot of the ways you work looks
completely ass-backwards to folks who are used to doing it the other way.

~~~
throwaway437812
Could you elaborate what is is that F# taught you, and how it changed the way
you work?

~~~
DanielBMarkham
I realize you're using a throwaway account, but this is really more like an
essay, not a HN comment (cue up snarky observations about the length of my
comments). There were four or five things that all came together at the same
time for me: lean startup, functional programming, TDD, and static page web
programming.

If you're interested, my contact details are in my profile. Happy to have a
call/Skype. It's an important topic.

~~~
baronseng
maybe a blog post and post it on hn?

------
USNetizen
Here's part of what I don't get - there are so many government organizations,
state and federal, jumping over themselves to provide funding, incubation and
support to IT product/app companies across the country claiming it creates
jobs when, as this shows, it is entirely possible to operate at incredible
scale nowadays with a mere skeleton staff. So why do so many government-run
and government-funded programs fawn over such companies when their ultimate
goal is job creation and these companies are designed to run with barely a
couple dozen people at most?

This has nothing to do with private VC's; I completely understand why they are
in the game. I'm talking about government programs or government-funded NGO
programs that provide tax incentives and other funding or support to startups
in the product/app space hoping it will create jobs. Maybe this isn't every
state, but where I live this is the case.

~~~
Balgair
Because most of those politicians think Google employs roughly the same number
of people that GM used to. Google has about 55k employees [0] and GM at it's
peak had about 350k employees [1], a 6 fold difference. GM also had high
competition in the marketplace with Ford and others, so the number of
autoworkers was likely in the millions for the country. Since the car and the
computer are synonymous for 'mental space' for most people, meaning they are
about equally used in terms of time and resources, the 2 industries are
considered roughly equal. Granted, there are a lot more tech companies out
there than auto companies even in the 80's heyday, so the number of workers
may be near parity.

Also, remember that Google and FB have put a lot of cash in to D.C. in terms
of lobbying [2]. FB is essentially using the start-up scene as their R&D
departments, occasionally having to fork out the moolah to buy WhatsAps and
other viable competitors.

[0] [https://www.quora.com/How-many-employees-does-Google-
have](https://www.quora.com/How-many-employees-does-Google-have)

[1][https://en.wikipedia.org/wiki/General_Motors#History](https://en.wikipedia.org/wiki/General_Motors#History)

[2][https://www.washingtonpost.com/politics/how-google-is-
transf...](https://www.washingtonpost.com/politics/how-google-is-transforming-
power-and-politicsgoogle-once-disdainful-of-lobbying-now-a-master-of-
washington-
influence/2014/04/12/51648b92-b4d3-11e3-8cb6-284052554d74_story.html)

~~~
shas3
GM manufactures their cars in their own factories. Electronics is largely made
by fab foundries and electronics assemblers like Foxconn. The entirety of a
Chevy Cruze is assembled by GM from parts sourced from other companies and GM
itself. Apple iPhones are assembled by Foxconn and others. Foxconn is supposed
to employ 300,000 people to assemble iPhone 5S in 2013 [1].

[1] [http://fortune.com/2013/11/27/by-the-numbers-how-foxconn-
chu...](http://fortune.com/2013/11/27/by-the-numbers-how-foxconn-churns-out-
apples-iphone-5s/)

------
perishabledave
For reference here is a talk they did about how they scaled WhatsApp and the
problems they faced:

[http://www.infoq.com/presentations/whatsapp-
scalability](http://www.infoq.com/presentations/whatsapp-scalability)

------
jamespo
I got my first ever spam on whatsapp the other day from someone who had never
been a contact, and there seems to be no way whatsoever to prevent it. So
maybe they should hire 1 more engineer to work on spam.

~~~
verytrivial
I installed whatsapp and the first three messages I received were spam, all
within about 30 minutes. I uninstalled whatsapp.

~~~
txrit
I've used it for at least 4 years, only got one spam message ever. I blocked
the sender. I didn't uninstall whatsapp.

------
amelius
> Why WhatsApp Only Needs 50 Engineers for Its 900M Users

Answer: because sending short messages from A to B is basically a solved
problem. There is even a programming language (Erlang) that was made with this
application in mind. The prototypical "Hello World" example for Erlang is a
messaging application.

~~~
oskarth
I'm sure you could do it in an afternoon, right? Just like anyone on HN can
make a "basic CRUD app" like Facebook in a weekend.

I really dislike the arrogant attitude behind these types of comments. Have
you worked at WhatsApp? Do you have any idea what they spend their days doing,
what their systems look like, what their requirements are like?

WhatsApp have close to a billion users, 50 engineers and they manage to
produce a near flawless experience. This is _not_ trivial, no matter how many
armchair critics claim otherwise. It used to be the case that people dismissed
FB as just another CRUD app ("not even written in RoR, but in PHP, yuck" ~
2005-2010), but given the tools they have been pushing the last few years
people seem to have stopped dismissing them as casually. Perhaps in a few
years people will stop dismissing WhatsApp as casually as well.

~~~
elorant
_Have you worked at WhatsApp? Do you have any idea what they spend their days
doing, what their systems look like, what their requirements are like?_

Do you? If yes, please enlighten us. Otherwise I don't see the point of your
objection. No one said that it can be done in a weekend. All that the parent
comment said is that the fundamentals of building WhatsApp are already solved.
He/she didn't dismiss it. Just because something isn't rocket science doesn't
mean it's not useful.

As for Facebook, people used to dismiss it not because it was seemingly too
easy to build but because they failed to see its usage. Long before it became
a platform it was pretty much a waste of time. Obviously anything with more
than a hundred MM users is difficult to maintain.

~~~
rdtsc
Not "parent" but search for Erlang Factory talks they talk about it. Here is
one from Rick Reed. He explains how the product works :

[https://www.youtube.com/watch?v=c12cYAUTXXs](https://www.youtube.com/watch?v=c12cYAUTXXs)

------
bendtherules
"WhatsApp doesn’t talk much about its engineering work" \- Wrong as hell. They
first talked about this in erlang conf about 3 yrs ago, in 2012. [Youtube
Link](
[https://www.youtube.com/watch?v=wDk6l3tPBuw](https://www.youtube.com/watch?v=wDk6l3tPBuw)
).

I hate how tech blogs take a thing thats quite well known and old and spins
into a "did you know?" post like it was just unraveled (Oh, maybe your staff
just heard about it, doesnt mean no one knows shit.)

~~~
rdtsc
Reading through comments, I think there is an undercurrent of slight envy or
resentment (perhaps unconscious) both towards WhatsApp and Erlang. We are
rational people so we'll never admit to that, so it comes out in other
objections "It is a simple system", "I could do the backend in a weekend"...

There are a lot of startup people here, who work at a startup, bet their time,
money, life? on one, and talking about a successful lean startup sold for 19B
probably bring feelings of "How come it is not my startup?".

And then for Erlang, it has different syntax, and have to think about
concurrency, functional programming, and it is hard for someone used to curly
braces. Maybe many tried it, but dismissed it, or bet (time, money, life?) one
something else so it is easy to say "But it is not about the platform or
technology". Because other conclusions might mean "heck, maybe I made a
mistake, I should have that up as well for my problem domain", nobody like to
feel that way. Of course, nevermind that engineers from WhatApp have mentioned
Erlang was one of their strategic advantages.

Managing 2-3M connections per machine means needing to have fault tolerance
(so Erlang is a good choice), but also means managing an order of magnitude
less machines! One of the deeper insights we can learn in general is that at
scale no matter how cool the concurrency mechanisms, how fast, and new, if
they don't go hand in hand with fault isolation then it will just be a very
fast system until it starts crashing and its throughput drops to 0. Of course
you can do fault isolation with separate servers and OS processes
(containers,VMs...) but in this case they do it at a finer grained level.

~~~
aianus
I resent them because Whatsapp is terrible, terrible software compared to
Facebook Messenger yet somehow made $19B.

1\. No web interface, I'm forced to use a tiny mobile keyboard

2\. Identity tied to my phone number which changes every month

3\. Ugly (subjective)

4\. My friends are already on Facebook, why have another service that does the
same thing but worse.

5\. No chat bubbles.

6\. No meme search

I could go on and on. They just got lucky and hit the lotto with foreigners
who couldn't afford decent phones and are now locked in to their platform.

Edit: formatting

~~~
sheepmullet
> 1\. No web interface, I'm forced to use a tiny mobile keyboard

Not a problem for most people. On the other hand it works well on pretty much
every phone out there. In comparison Facebook messenger runs terribly on my
iPhone 4. I'd hate to try and use it on a lower end android phone.

> 2\. Identity tied to my phone number which changes every month

I've had the same phone number for over 10 years. Everybody understands and is
familiar with phone numbers and have been for a long time. My 70 year old
mother has no problems using whatsapp. Many more people have my phone number
than have my email address.

> 4\. My friends are already on Facebook

Good for you. Many of my friends aren't. Most of my family and relatives
aren't. I'm thinking of leaving it myself as it's just a big time waster.

> I resent them because Whatsapp is terrible, terrible software compared to
> Facebook Messenger yet somehow made $19B.

Except it's really only bad for your particular use case. For most people
Facebook messenger was late to the party and doesn't have any real draw.

Facebook messenger still doesn't have basic functionality I want. For example
if I scroll to a particular point in a chat conversation and later come back
to the conversation FB Messenger has not kept my scroll position. Basic basic
usability problems.

~~~
aianus
> For most people Facebook messenger was late to the party and doesn't have
> any real draw.

Facebook Messenger has been around since the launch of Facebook. I've
personally been using it over 7 years in various forms. WhatsApp is the one
that was late to the party.

> On the other hand it works well on pretty much every phone out there.

Of course it does, because it has no features. You can't send money with it,
you can't send stickers or GIFs, you have to physically open the app to send a
message, it doesn't have link previews, and so on and so forth.

~~~
sheepmullet
> Facebook Messenger has been around since the launch of Facebook. I've
> personally been using it over 7 years in various forms.

And it was a crap experience on mobile until it was pulled out of Facebook
into its own app.

> Of course it does, because it has no features.

It has the most important features for good individual and group chat.

> you can't send stickers or GIFs

That's a feature.

> you have to physically open the app to send a message

I'm not sure what that means. I have to open the FB Messenger app too? Maybe
it's different on android.

> it doesn't have link previews

Again another point in whatsapps favor. I hate link previews in FB Messenger.

> You can't send money with it

Not really much of a feature because of all the setup involved. It's got
potential and if it could tie in with the host operating system or payment
providers etc on mobile it could be really great.

------
ilitirit
Worth reading, especially the stuff that is linked:

[http://highscalability.com/blog/2014/2/26/the-whatsapp-
archi...](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-
facebook-bought-for-19-billion.html)

~~~
oskarth
This was a much better article. I submitted it to HN:
[https://news.ycombinator.com/item?id=10226317](https://news.ycombinator.com/item?id=10226317)

Perhaps the mods could merge/replace this (pretty bad) Wired article?

------
BuckRogers
Yup, Erlang. If I move on from Python as my main programming language of
choice, it would be to Erlang. So many are enthralled by Go and I never saw
it. The Google association helps for those that idolize the corporation.
Others feel Java is all we need or C# is all we need. Or JS and C++ and call
it done. Some want to JS all the things. Erlang is a bit niche but it's
indispensable like C, and Python is just such a great gen purpose platform.
The most value is in the C/Python/Erlang triforce.

------
kozukumi
WhatsApp seems to be a great example of designing your system with the best
tools available. To go from nothing to pretty much being the de-facto modern
replacement for SMS in a couple of years is an amazing achievement.

I really need to sit down and spend some time properly learning a functional
language. I can hack my way through Haskell code but I do it in a very non-
Haskell way. I really need to change that. My primary language is C++ and I am
open to any recommendations for what direction to go in :)

~~~
fsloth
For Erlang, there is the book by Joe Armstrong:

[https://pragprog.com/book/jaerlang2/programming-
erlang](https://pragprog.com/book/jaerlang2/programming-erlang)

The problem with learning functional languages is finding pragmatic learning
materials.

I would claim F# is currently the most approachable functional language (just
my opinion but I have at least toyed with the other mainstream offerings).

My background is also in C++. The F# materials listed helped me personally to
develop as a programmer and have made approaching other languages like Erlang
easier.

Sestoft's programming language concepts:
[https://www.itu.dk/~sestoft/plc/](https://www.itu.dk/~sestoft/plc/)

Expert F# 3.0 contains lots of short snippets and examples:
[http://www.apress.com/9781430246503](http://www.apress.com/9781430246503)

Flying frog consultancys materials are steep in price but I found them worth
my dime (Visual F# 2010 for Technical Computing and the materials in The F#
Journal)
[http://www.ffconsultancy.com/products/index.html](http://www.ffconsultancy.com/products/index.html)

~~~
maze-le
> The problem with learning functional languages is finding pragmatic learning
> materials.

This is kind of true. But there is a nice community around erlang. I always
found good, valueable input at erlangcentral.org and stackoverflow.

------
kartan
Because adding more engineers will make the development slower
([https://en.wikipedia.org/wiki/The_Mythical_Man-
Month](https://en.wikipedia.org/wiki/The_Mythical_Man-Month)).

~~~
tams
WhatsApp is a service in production, not a project behind deadline.

~~~
fsloth
Yes, but the reason more people create overhead in a late project are the same
why they create overhead in an organization.

------
rwj
There are more than 50 engineers working to support WhatsApp. A complete count
should include all of the work that was out-sourced, such as engineers at AWS
(or whomever is supplying the computing), the engineers for Erlang, etc. If
the government builds a bridge by using a construction company, no one says
the government is capable of building a bridge with zero engineers.

The app is still a remarkable achievement. Those engineers have done a
remarkable job integrating everything (plus their own business logic on top),
but they are also leveraging a great deal of work done by others.

~~~
hansolo669
> If the government builds a bridge by using a construction company, no one
> says the government is capable of building a bridge with zero engineers.

However I doubt people pay much attention to the companies quarrying rock to
make concrete, or the engineers designing the axles of the trucks. Any large
project is a team effort, but to count tangentially related projects as
partitions of another ignores the achievements of those projects (i.e. AWS and
Erlang are massive achievements that have, and will, exist regardless of
WhatsApp).

So, WhatsApp has 50 engineers modulo any dedicated support staff.

------
paradite
On a side note, there is a Chinese Internet messaging software called QQ
developed by Tencent (which later developed WeChat). it had hundreds of
millions of users about 8 years ago, and it runs on multiple platforms,
including Nokia, Android, etc.

So the point is that it is not surprising for a similar app to be developed 5
years later by a small group of people.

[https://en.wikipedia.org/wiki/Tencent_QQ](https://en.wikipedia.org/wiki/Tencent_QQ)

------
feld
They didn't cover that they use FreeBSD and are handling 2 million tcp
connections per server which is not a trivial task

~~~
swills
There is a brief mention of FreeBSD in the article, but more would have been
nice.

------
codeisawesome
"Code in Parallel In using Erlang, WhatsApp is part of a larger push towards
programming languages that are designed for concurrency, where many processes
run at the same time."

Concurrency is not parallelism! /whacks author over the head with a Go manual

~~~
acconsta
Yeah, but what's the best way to explain concurrency to a general audience?

~~~
codeisawesome
Let me give it a non-journalistic stab:

"A method of programming where you can safely allow many different, but
related computations to occur simultaneously to arrive at a desired result.
Thanks to the "safety" aspect, these computations can be automatically
processed in parallel (at once) by many thousands of computers to increase the
speed of the program, but no collisions of data processing will occur. In
other words, no two parallel processes will work at cross purposes as a result
of parallelization - when the code is written correctly and concurrently."

------
alkonaut
Language/platform choice explains why they need 50 engineers rather than 100.
The trivial scope of the application explains why it's 50-100 and not 5000.
It's a messaging service! If they weren't extending their application by
developing new functionality then 50 even sounds like a lot (assuming they
don't run any of their own hardware).

~~~
koffiezet
Don't forget they also have to develop the front-end applications, which
include iOS, Android, web, ... That will probably be the largest part of their
engineering effort.

------
hudell
I'll never understand why so many people prefered WhatsApp over anything that
already existed previously and even worked better.

~~~
piyushpr134
To elaborate, there are many reasons 1\. They have a no ads policy which is a
really godsend thing when other messengers had stickers, games and just plain
annoying ads. 2\. The ones that did not, for ex Google etc were not really
made for phones. They understood the importance of using your contacts as
social graph really really well. 3\. They almost never experienced outages.
4\. Had sensible defaults. 5\. Their apps are lightweight (memory and cpu
wise) compared to amount of time one spends on it. 6\. Had no api and
generally very low spam compared to sms. SMS has really high spam in the
developing world. 7\. Rich media compared to SMS and no character restrictions

I think west really hates it as most of the people there have iphone and they
had this $1 signup which made it unpopular there. They never had that on
android which made it a darling of non-west

~~~
aianus
Facebook Messenger has none of those problems and is better in every way than
WhatsApp.

------
hashin
This articles doesn't look that convincing to me. I has no technical details
to prove that Erlang was the major reason why WhatsApp was scalable. I think
the low number of engineers has more to do with the simplicity of their
product than with this particular language choice.

~~~
biokoda
WhatsApp engineers themselves said Erlang was the main reason they were able
to scale as well as they did.

~~~
jamespo
Didn't they get a big head start from ejabberd? I hope they gave them some
stock.

~~~
biokoda
Yes I think so

------
mcintyre1994
I have to say I was expecting this to be a little older - it's really
impressive that Facebook have kept the culture and tiny headcount of that
team.

I doubt it's ever a good idea for a company like Facebook to do layoffs but I
wonder if there are any lessons from the WhatsApp team for a company that size
- their intern cohort looked way bigger than 50 and they seem to have almost
50 offices [0]. Obviously a very different product and much more complex but
it's interesting to wonder what the same product led by Jan Koum would
hypothetically look like.

[0] [http://newsroom.fb.com/company-info/](http://newsroom.fb.com/company-
info/)

------
apexkid
I wish there were some good online resources around Erlang. Most of the
content is very theoretical and not explanatory. Its hard to find
implementation of common algorithms like Sorting, graphs, link-list and other
dataStructures in erlang.

~~~
amorphid
I haven't taken the Erlang plunge yet, but if I did, I'd probably try starting
with Elixir, a language written in Erlang. There is a growing amount of buzz,
community, tooling, etc. A basic understanding of Elixir would probably make
Erlang a lot easier to understand.

~~~
clessg
> A basic understanding of Elixir would probably make Erlang a lot easier to
> understand.

It does, can personally confirm.

------
KuhlMensch
The language and culture of simplicity are great aspirations, all companies
should look at doing this. However, not all companies are going to get such a
huge benefit, as not all companies operate within a narrow problem domain.
INSTAGRAM is another similar case: relatively simple solution (share photos
with limitations), uses only single language PHP (at least since last I
heard).

I mean this is partly a reflection of the teams brilliance, and the foresight.
But its also that the operational cost is relatively low compared to a payroll
system that needs to abide by government regulations and reporting
requirements (for instance).

~~~
joeyspn
PHP? They're known for being a huge Django (Python) success. Or did fb
converted them to HipHop?

~~~
monkey_slap
Still Python here

------
joelthelion
I think part of it is their awesome product design: their app has few
features, but they are well thought out and enough to make WhatsApp very
useful. Fewer features mean less people needed for development and operations.

------
wslh
I would love to see a list of these 50 engineers, their skills and how they
are organized. It provides guidance for other teams (even if part of their
software is written in Erlang!).

~~~
aet
Type "Whatsapp" into LinkedIn and you will get some idea, but probably biased
towards more junior people. I agree would be interesting to see.

~~~
wslh
Beyond the individual roles it would be nice to know about their organization.

------
KirinDave
For reference, Level Money sold to C1 and had 1 backend engineer, 1 android
engineer, and 1 ios engineer.

This scale isn't unusual at all. Really it's a lot more about feature
development. The smart way to build most mobile startups is to use managed
providers like google app engine, AWS Container Service, or Heroku. Between
that and how not-bad both Android and iOS are to develop for these days
(compared to their history), it's possible to get your engineering team size
pretty small.

~~~
acconsta
>This scale isn't unusual at all

How many apps are used by 13% of the world population every month?

~~~
KirinDave
Depending on your architecture, this difference is often immaterial. That's
what people don't get.

When architected correctly, the actual amount of code to be written and
maintained is quite low. And an easy way to jump-start that architecture is to
have other people who have amortized the cost into a service model do that
work for you.

------
ghshephard
"Enlightened Product Management" is the phrase I've heard to describe
WhatsApp's success in keeping their engineering requirements to a bare
minimum.

------
ericjang
Question for HN: what is your estimate on the minimum man/womanpower needed to
scale a product/service up to 10K users? 1M? 900M? For instance, what fraction
of Facebook's work is related to making things scale? How much effort does a
smaller company like Quora put into scaling for users? I imagine PaaS/IaaS
services like Google App Engine and Amazon Elastic Beanstalk help a lot but
I'm curious as to what you think.

~~~
temuze
Depends on the product!

Are you making something that searches the entire internet for relevant
documents based on a custom query? Or are you sending a picture to another
phone that goes away in < 5s? Or is this a blog?

If you're talking "minimum" then theoretically it could be 1 person. It's very
easy to scale static pages. Heck - you could host a static page on S3.

~~~
acconsta
Sure, but what's the most popular application hosted on AWS? Does it approach
900 million users?

~~~
joeyspn
It was probably Instagram before they moved to FB datacenters, with around
150-200 million monthly users ...

[http://instagram-
engineering.tumblr.com/post/89992572022/mig...](http://instagram-
engineering.tumblr.com/post/89992572022/migrating-aws-fb)

~~~
acconsta
Maybe Reddit now then? 200 million monthly uniques:

[https://www.reddit.com/about/](https://www.reddit.com/about/)

------
hokkos
Because they kept their product simple and focused, it does a few things well.

The Erlang thing is nice a nice fit, but mostly because they leveraged an open
source product.

------
e0m
What I want to hear more about is how they got those users in the first place.
Supporting 900M users might be easier then acquiring 900M users.

~~~
bra-ket
great product + word of mouth

------
pbreit
I think this is the key line: “The number-one lesson is just be very focused
on what you need to do”.

My sense is that WhatsApp spends all of its energy on the very core product
offering and not a bunch of superfluous activities. I don't think the language
choice has much to do with it unless there's some sort of "keep it simple"
thinking that the languages somehow exude.

------
guj11
Does anyone know if WhatsApp uses Scrum (or similar methodology)? Also, do
they do pair programming?

My company has recently learned about scrum and pair-programming, that seems
like the only way to success to them. So, it would be nice to know how other
successful companies are organized and develop software.

~~~
RangerScience
Only comments as to a part of your question, but: "Employees rarely attend a
meeting". Stand-ups are still meetings.

------
calgoo
I heard that they have the philosophy of "you build it, you own it", meaning
that anything you build, you are also responsible for running. So if you don't
want a call at 3am, make sure your code works correctly.

------
vinceyuan
I did know WhatsApp uses Erlang which supports hot deployment. But after
reading this article, I still don't know _Why WhatsApp Only Needs 50 Engineers
for Its 900M Users_.

------
rurban
Why would you need more than 3 engineers?

This article should talk about what the 47 others are doing.

------
mahyarm
The reason why whatsapp can keep it small is because they decide features
based on engineering cost first. Does the designer want a feature or UX design
that would be a pain in the ass to create with this UX kit? Is there a
probably uglier equivalent that would take 1/10th of the engineering time? It
gets shot down and we do it the engineer-simpler way.

One UX example in the iphone app is the action sheet that pops up when you
want to send photos or your location. While in facebook it gets embedded in a
fancy keyboard drawer. The action sheet version probably took about 5 hours
total for all of it, while facebook messenger's version probably took 500
hours minimum.

Another example on the backend is how they are relatively stateless, do not
keep chat histories, do not have real multi-device w/ one account capabilities
and so on. Each of those features would increase engineering and machine cost,
sometimes significantly.

~~~
kcorbitt
I totally agree with what you're saying. But remember that they got to 900M
users with this approach. Obviously users liked something about what
Whatsapp's philosophy produces.

------
linkydinkandyou
Our company is solving a similar large-scale problem with very few engineers.
And our answer, too, is Erlang! (And also, a few very smart, highly
experienced, people with Math graduate degrees.)

~~~
davidw
So... what are you doing? Link?

