
Scaling Erlang Developer Experience at WhatsApp [pdf] - anuragsoni
https://codesync.global/uploads/media/activity_slides/0001/03/f2292f201aa6b04db8c4e0b9cfa191dd07c9ee14.pdf
======
amgreg
What I like most about some experienced engineers who have been at companies
from the beginning is that they realize the needs of a small-team, cash-
constrained, rapidly growing company are different from that of a dozens-of-
teams, complex, but also rapidly scaling enterprise. The devil is in
understanding when and how to transition your tooling, architecture, and —
dare I say it — attitudes, from the one to the other.

I am curious to hear more from these engineers; how decisions were made about
transitioning/scaling; and what decisions were made.

~~~
thdrdt
There are things that could (should?) be done from the start. Most has to do
with automation. For example automated build tools. Maybe you don't want
automated unit tests in the beginning but those parts can be added later
without changing the company's culture and stucture too much.

Or automated invoices. It is much easier to change this than changing a co-
worker who manually created all the invoices.

~~~
spinningslate
> Maybe you don't want automated unit tests

Interested why you'd say this, can you elaborate? Do you mean tests in general
or unit tests specifically?

I can't imagine automating a CD pipeline without having some automated testing
as an integral step. If you don't have automated tests, how do you remain
confident that you haven't introduced bugs/regressions?

Even in a small codebase that's easy to do - especially when you're evolving
quickly.

Like I say: not tacit criticism, genuinely interested.

~~~
ralston3
I think the key point in the OP’s statement is “early on”

I think unit tests (or just more generally a test suite) is invaluable. But
when you’re starting out small (1-3 person dev team) and the API is changing
so rapidly (we learned something this week we didn’t know last week so that
new service now only does 1 thing instead of 5 things), tests can _really_
slow down development. And as mentioned by the OP as well, adding unit tests
(or a test suite in general) shouldn’t be too hard once you get a stable API

I might be preaching to the choir so to speak, but that’s just my $0.02

------
kornish
One of my favorite talks is by Rick Reed about scaling Erlang at WhatsApp.
What an absolute savage. He flies through an articulate and in-depth
curriculum on system performance and bottleneck mitigation.

The talk is called "That's 'Billion' with a 'B'" and makes for a great
lunchtime watch:
[https://www.youtube.com/watch?v=c12cYAUTXXs](https://www.youtube.com/watch?v=c12cYAUTXXs)

~~~
strmpnk
I remember that year. I was giving a talk at EF during the same time slot but
the schedule originally had me in the large room and they had a much smaller
one.

When the news of the acquisition hit, everyone wanted to see the WhatsApp
talk. The organizers knew this so we swapped rooms. So, I started my talk by
asking if anyone in the room was here for the WhatsApp talk and told them they
could quietly leave and I wouldn't mind and a bunch of people got up.

Heheh. I don't blame them. I didn't really like my talk and Rick Reed is very
good at what he does and the talk is no exception.

------
an_opabinia
Erlang is the Actors-model version of Greenspun's tenth rule of programming:

"Any sufficiently complicated server backend contains an ad hoc, informally-
specified, bug-ridden, slow implementation of half of Erlang."

Is it really the end-all-be-all model of server architecture? What are
microservices, if not Actors Clunkified? "Serverless" functions, if not the
simplest actors in disguise? Docker, actors except over IP addresses and
clunky APIs?

The real takeaway: Valuable stuff is stateful. You know you're doing valuable
work for a real human end-user if you are dealing with tricky state problems.
Extracting money out of that is a bunch of developer-productivity-limited
business concerns. It seems actors are a good model to solve that.

~~~
davidw
That quote regarding Erlang is attributed to Robert Virding.

------
stevencorona
I absolutely love the BEAM and Elixir, but one of my common complaints and
source of errors is the lack of compile-time type sanity checking, so I'm
excited to see movement in this direction. Dialyzer is sorta okay, but the
speed, cryptic errors, and syntax have kept me from fully embracing it.

~~~
findjashua
Gleam may be what you're looking for:

[https://gleam.run/](https://gleam.run/)

~~~
callamdelaney
This isn't ready for prime time yet iirc.

~~~
sodapopcan
It's not but is in constant development. The author is very active on
elixirforum.com

------
AshamedCaptain
I find it funny they say Whatsapp "chose" Erlang. They "chose" Erlang because
ejabberd was written on it, and they chose ejabberd because of a Jabber
mailing list suggestion. That's about it...

~~~
chii
are you saying that the choice was purely due to luck? Would whatsapp have
been as popular and successful had it not been in erlang but some other
language/stack?

~~~
krn
> are you saying that the choice was purely due to luck?

What he is saying is, that they chose the best open source tool available for
their use case at that time, and not a programming language.

> Would whatsapp have been as popular and successful had it not been in erlang
> but some other language/stack?

I would argue, that they might not have been as efficient in terms of
computational resources and the number of technical employees needed, but they
would still have succeeded even if the back-end was implemented in Java.

Because what all WhatsApp users actually interacted with were the clients –
and they were not written in Erlang.

------
tiffanyh
> "We are working on a prototype, open-sourcing in November"

I'm curious to see how Elixir integration with such static type system would
work.

~~~
seanclayton
I absolutely love Elixir and use it all the time, I really just wish it had a
type system (that's not the slow behemoth Dialyzer). I understand the
difficulty of typing processes and I don't envy the task, but as a user of the
language it would make it _that_ much better to work with! Just imagine
working with a language like Elixir that has the type system of F#!

~~~
acjohnson55
Sounds like Scala / Akka to me

------
derefr
Last I heard (years ago), I though that WhatsApp's backend was being rewritten
in some other language, because Facebook didn't think maintaining an Erlang
codebase was tenable. Did that change?

~~~
ramchip
Perhaps you're thinking of Facebook chat?

[https://www.quora.com/Why-was-Erlang-chosen-for-use-in-
Faceb...](https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Facebook-
chat)

[https://www.quora.com/When-did-Facebook-switch-away-from-
usi...](https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-
for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with)

------
gautamcgoel
Can someone explain what Erlang offers that Go does not? Both have great
support for concurrency, but Go has type checking and a more familiar syntax.

~~~
masklinn
* Erlang builds _reliability_ into the language and runtime through concepts of liking, supervision, …

* Erlang is a much simpler language

* Despite lacking a static type system, Erlang's functional bend, immutability, SSA and pattern matching makes for a rather safe language

* Erlang has always been fully preempted, it didn't have to wait until 2020 for that to happen

* Erlang also builds distribution in, though it's "trusted distribution" so probably of fairly limited use at large

* Erlang uses shared-nothing concurrency, greatly limiting side-effects across processes, which again helps with reliability: there is no action at a distance in sequential erlang, no third-party can come in and mess with your state

* when dealing with binary data, binary patterns are absolutely amazing (and rather unique)

~~~
coder543
> * Erlang is a much simpler language

Go is widely considered to be a very simple language, often too simple, so I
find this claim interesting.

> * Despite lacking a static type system, Erlang's functional bend,
> immutability, SSA and pattern matching makes for a rather safe language

This is fine stuff, but I would rather have a type system, so this is just
making up for a deficiency with some other stuff, so it's really just a
sidegrade, not so much an upgrade or a downgrade.

> * Erlang has always been fully preempted, it didn't have to wait until 2020
> for that to happen

With extremely rare exceptions, Go has _always_ been fully preempted from a
developer experience point of view. The fact that the underlying
implementation was using function calls as yield points simply didn't matter.
The fact that it is fully preempted now means it's kind of weird to still hold
it against Go that they are doing the same thing as BEAM now.

It is _done_ , this is no longer a bullet point.

> * Erlang also builds distribution in, though it's "trusted distribution" so
> probably of fairly limited use at large

Go has _always_ "built distribution in" by offering to build single static
binaries. If anything, BEAM languages are _harder_ to deploy.

> * Erlang uses shared-nothing concurrency, greatly limiting side-effects
> across processes, which again helps with reliability: there is no action at
> a distance in sequential erlang, no third-party can come in and mess with
> your state

I agree that immutability is an advantage. In general, third parties can't
come in and unexpectedly mess with your state in Go either, as long as you're
either maintaining your state local to a function or keeping your global
variables private to each package.

Keep in mind that Go's package imports must form a tree with no loops. Third
party packages literally cannot see your code or any of the exported global
variables, since you are importing them, and not the other way around.

Go does not offer any way for packages to reflect over the whole program's
state in memory, so there's no way for third party packages to go hunting
around in memory and playing with your values, which would be stupid, unless
they use "unsafe" to start poking at random memory addresses, and I'm certain
you can do something equally dumb in Erlang with a sufficient amount of
effort. It's just as crazy to think of doing something useful that way in Go
as it is in Erlang.

> * Erlang builds reliability into the language and runtime through concepts
> of liking, supervision, …

This is literally the only compelling talking point I've seen for Erlang/BEAM
vs Go.

However, I've written large Go applications with distinct services running
within the same binary, and it is possible to make extremely reliable systems.
Go's context package allows you to build trees of processes that you want to
be able to cancel, including cancelling specific sub-trees without affecting
adjacent trees or parent trees.

The single biggest footgun with building reliable systems in Go is that it
_prefers_ to crash early. So, every Goroutine that you launch, you must make
sure that the first thing it does is catch any panics to avoid taking down the
entire program, including unrelated Goroutines. When correctly written, a
human will be immediately alerted to the bug that caused this panic (which
shouldn't ever happen anyways), and the program will continue operating
normally in the mean time until a human gets a chance to push out an update to
fix that bug.

In practice, enforcing just one rule (Goroutines must catch panics) in code
reviews is not that hard, and it isn't common in my experience for most code
to arbitrarily launch Goroutines -- Goroutines should be managed.

In return, Go offers much better performance than BEAM, as well as most of the
practical benefits of BEAM that I've seen discussed online.

I think Rust is a better language than either Go or any BEAM language, but the
absence of a preemptive concurrency model like Go and Erlang offer just makes
it hard for me to enjoy the prospect of using Rust for general purpose network
services.

Once Go adds generics in the next year-ish, that should solve my biggest real
annoyances with the language. I really look forward to being able to package
up a number of the concurrent patterns I write into reusable packages, which
will largely eliminate once and for all the need for me manually spin up a new
Goroutine in the code that I'm writing and potentially forget to catch any
panics.

~~~
Jtsummers
> > * Erlang also builds distribution in, though it's "trusted distribution"
> so probably of fairly limited use at large

> Go has always "built distribution in" by offering to build single static
> binaries. If anything, BEAM languages are harder to deploy.

I suspect GP meant distribution in the distributed computing sense, not the
deployment sense. What, baked into Go not via a library, handles multiple Go
_programs_ communicating within the same physical node or across multiple
physical nodes? Erlang gives you the _same_ communication method whether
distributed or within a single node, Go does not (that is, Go channels are
internal to a specific program, they cannot be referenced from different
programs or computers).

If they did mean it in the deployment sense, I'd still argue it's a point in
favor of Erlang. Two nodes communicating can execute code on the other.

~~~
coder543
Yes, I definitely misunderstood the point being made there.

> What, baked into Go not via a library, handles multiple Go programs
> communicating within the same physical node or across multiple physical
> nodes?

Go does have an RPC package built into the standard library, for what it's
worth, but I know it's not the same thing as the runtime transparently
stitching distributed machines together under the hood.

------
pjmlp
> shift to modern languages with integrated tooling, e.g. Erlang competition:

>

> C++, Java => Go, Rust, Kotlin

My first C++ IDE was Turbo C++ 1.0 for MS-DOS in 1993, and Borland, Zortech,
Symatec, Microsoft, IBM, and plenty of others had well known C++ IDEs.

Many of the ideas of modern C++ IDEs is basically rediscovering the work done
by Lucid and IBM with their Smalltalk like capabilities for C++ tooling in the
late 80's/early 90's.

Java is well known for being one of the languages with best IDE support out of
box. It didn't start like that in 1996, but the last 25 years have been good
to it.

Go and Rust are yet to have similar offerings and Kotlin, well it is a JVM
language and profits from the Java eco-system and Google's push to replace it
on Android due to non-technical reasons.

Maybe better examples should have been provided.

~~~
saagarjha
Java is interesting because writing code in it is generally absolutely
miserable without an IDE.

------
fbn79
Working in a small Company always let me astonished learning things about
unicorn apps. But really an application like Whatsapp need 1000+ developers!?
I can think about teams about android, ios and web app. X2 if you think about
teams for business oriented services. But 1000+ developers for what?

~~~
jamestimmins
Just some of the needs at that scale:

    
    
      Fraud
      Security
      Integrations
      Infrastructure
      Data
      Accessibility
      Growth
      I18n
      API
    
    

In addition, there won't be individual teams for android, ios, etc. Those
domains will be broken across many task-specific teams.

------
AzzieElbab
I am curios why they need 1K people instead of 10? What changed on the server
side of WhatsApp?

~~~
abrolhos
I understand that the 1K+ engineer number comes from Facebook as a whole, when
mentioning the use of Hack as a typed PHP.

More like to give a perspective on the transition on how to come from a small
company into a huge one.

~~~
user5994461
WhatApps got audio and video calls after the Facebook acquisition.

Usage increased by a few orders of magnitude too.

~~~
stjohnswarts
Yeah I think people get usage confused with number of users.

------
ramon
I think the concepts are very interesting of Erlang but I still don't
understand it fully. And also about the multi-node os only an internal network
scenario is not like a cloud mesh thing. You would need docker.

------
btbuildem
> some {T} | undefined

Can we please avoid TS pitfalls?

~~~
phowat
Would you mind expanding a little bit on why this is bad ? I'm fairly new to
TS and find myself doing it a lot.

~~~
angio
The advantage of the Option type is that you don't need to check for
nullability, you simply pattern match on Some(_) or None. A type of `some{T} |
undefined` means that the expression can have values `Some(_)` or `None` or
`undefined`, which defeats the point of the Option type!

~~~
masklinn
Except Erlang doesn't have a none. Nor an undefined for that matter. So `some
{T} | undefined` means exactly that, there is no underlying null pointer which
could sneak up on you.

`undefined` here is not built-in type or value of the language, it's a
standard atom like any other.

So are null, nil, value or false (although false does have a special status in
that it's the customary output of boolean operations alongside true).

~~~
angio
I'm aware of that, I was explaining why `some{T} | undefined` is problematic
in Typescript. If you have a signature with both option type and undefined
it's a signal of a code smell.

~~~
mbo
But `some{T}` isn't an option type, it's a member of the option type!

