
Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD users [video] - adamnemecek
https://www.youtube.com/watch?v=TneLO5TdW_M#
======
wallflower
To me, the more fascinating fact is that Erlang enabled their massive ability
to scale.

[http://highscalability.com/blog/2014/2/26/the-whatsapp-
archi...](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-
facebook-bought-for-19-billion.html)

~~~
Ixiaus
What's more fascinating is that erlang underpins many more staples of the
internet than most geeks even know. WhatsApp is just one 19B example.

~~~
pjmlp
Not only the Internet.

A few German game studios are using it on their backends.

~~~
marktangotango
Physics in Erlang? Game scripting engine implemented in Erlang? What types of
games are we talking about here?

~~~
pjmlp
I wrote backend. Physics and game scripting belong on the client side,
usually.

[http://de.slideshare.net/wooga/erlang-the-big-switch-in-
soci...](http://de.slideshare.net/wooga/erlang-the-big-switch-in-social-games)

[http://sdtimes.com/enterprises-can-learn-from-game-
developme...](http://sdtimes.com/enterprises-can-learn-from-game-development/)

------
perlgeek
This feels like mixing of levels to me. The Internet traffic of nearly every
user will run through a Cisco or Juniper router. Does that make us all IOS or
JunOS users?

~~~
rdtsc
It will and if Junos or ios stop working so will the internet to a large
extent. It speaks to reliability and scalability aspects.

You could say the same about erlang for example, it used by WhatsApp and in
many internet-cell network gateways.

------
justincormack
Netflix is the other FreeBSD based service that many people dont know about.
While the control plane is the much hyped Linux on AWS stuff, the actual
streams are FreeBSD boxes in ISPs[1].

[1]
[https://openconnect.itp.netflix.com/software/index.html](https://openconnect.itp.netflix.com/software/index.html)

~~~
tachion
There's also Juniper, EMC2, Apple, Yahoo (back in the day, at least), Sony
PlayStation and many, many others.

~~~
justincormack
I would only really count Juniper in that list, building on and with FreeBSD.
Apple take some bits but their product is not really FreeBSD (alas) and they
dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.

~~~
feld
EMC Isilon still uses FreeBSD. There are employees who are also FreeBSD
developers. They're working hard on giving back changes so they can jump to a
much newer FreeBSD release.

Yahoo's FreeBSD isn't dead; YBSD! still exists too.

------
praveenster
Since Whatsapp doesn't have client facing web page, but only api's used by the
client apps, are their scalability requirements different from those that have
a web interface as well?

~~~
Demiurge
messages are probably much smaller, but more numerous and latency has to be
'realtime', but also client-server-client juggling. so, to me this means they
need to respond faster, more often, and deal with more sharing than a website.

------
jambo
I found the pragmatism and sense of deferring until the least responsible
moment—especially the speaker's view of what that moment is—very interesting.

In the Q&A "What do you use for configuration management?" "<slight laugh>
`make`", going on to say they don't have enough node types or a large enough
cluster to do automation yet.

------
general_failure
Not sure the title is correct. After all, the people use WhatsApp through
Linux (Android), Symbian or whatever and not FreeBSD.

~~~
adamnemecek
The servers were (are?) FreeBSD.

~~~
tachion
The servers are FreeBSD 9.x. After Facebook acquisition they've tried
migrating it all to Linux but quickly discovered that Linux cant cope with
that amount of traffic and they decided the FreeBSD is the tool to be used
here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers
to help them bring the Linux TCP/IP capabilities in par with FreeBSD.
Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS
upgrade.

~~~
atmosx
Do you have any idea why Facebook would rather develop a TCP/IP Stack for
Linux to match FreeBSD's, instead of further evolving FreeBSD's stack?
Licensing issues?

~~~
justincormack
How could licensing issues be a problem?

I think people prefer a monoculture, even though the security idea of having
multiple software stacks is appealing, thoroughly testing issues on two OS
stacks is a lot more work, especially at Facebook scale.

Presumably they are tied into Linux for a lot of their internal
infrastructure.

~~~
Perdition
The main security threat these days isn't viruses or worms so a monoculture
makes no real difference.

In fact running multiple operating systems is more likely to lead to a
misconfiguration issue which allows an attacker into your network.

~~~
justincormack
If someone getting into your network is your threat model then, well, you have
already been owned. If you want to have resilient services where you can swap
them out if there are security issues, a monoculture is not a win.

------
sandGorgon
Correct me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to
be zero-copy sockets. This should be solved in Linux when kdbus is turned on
(replacing the current IPC : D-Bus).

------
glasz
just throwing myself in here: Starve [1] also runs on FreeBSD.

[1] [https://www.starve.io](https://www.starve.io)

------
sandGorgon
if one HAD to choose a non-erlang stack (for whatever reason), what would you
choose ?

Go, Nodejs or Java - I'm mainly concerned about clustering, because I think
that is what gave the Whatsapp devs their ability to scale(without having to
do too much re-engineering).

Or can the clustering be moved to something like the queuing layer ?

~~~
pjmlp
Java, of course.

There are lots of JVM implementations to choose from, many with high quality
code generation (both AOT and JIT) and GC implementations.

It has VM and cluster monitoring tools only comparable by .NET offerings.

The majority of big data solutions are built on top of it.

There are lots of libraries to choose from.

If Java 8, the language, isn't your thing, there are lots of modern languages
to choose from.

~~~
sandGorgon
I wsa hoping you wouldnt say that :(

Java high performance clustering is... intimidating. It is too enterprisey.
From my (little) research, it seems that one would need to build this ground
up on top of Netty or Undertow.

~~~
marktangotango
Care to elaborate? Load balancing to a bunch of http servers, sticky sessions,
or store the session to a db, or the best option is no session data,
memcached, its all pretty straight forward?

------
dschiptsov
Showing FreeBSD/Erlang stuff that just works and cope with substantial amounts
of traffic is not politically correct, because it will cause a severe pain of
cognitive dissonance to the Docker/JVM/Node/Cloud crowds..

Lets chant together: JVM is the best VM in the world with millions of man-
hours invested in optimizations. Servlet/JSP are mature, battle-tested,
enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-
scalable internet services. Everything must be a containerized micro-service,
orchestrated by Docker/CoreOS/systemd.

~~~
pjmlp
It is worse than that, where you would offshore development like on the other
cases? :)

