
Which companies are using Erlang, and why? - lelf
https://www.erlang-solutions.com/blog/which-companies-are-using-erlang-and-why-mytopdogstatus.html
======
ihuk
We developed Online Charging System in Erlang that served couple million
subscribers for close to three years. I found the whole experience fairly
terrible.

Erlang is nice enough language and I don't mind the syntax but it's also kinda
cumbersome and verbose at times. For example, adding `true -> ok` to every if
statement gets old fast. Similarly, Erlang/OPT is a nice platform but some
parts are fairly bad. Looking at you `inets/httpc`.

But the really big problem was the whole ecosystem. Build process was
needlessly complicated. Surprisingly enough multicore support was not great.
Performance was not that great. It seemed like a lot of libraries were
abandoned after 2013. I could go on.

We ended up rewriting the whole thing in Java and, so far, it worked out
great. And after Java 8, Java the language is not so bad. I still have
somewhat found memories of Erlang but I don't miss it.

~~~
e_proxus
Not to refute your experience, but "adding true -> ok to every if statement"
sounds like an anti-pattern. A couple of notes to elaborate:

1\. If-statementes should rarely be used in Erlang because case is preferred

2\. If you're returning ok everywhere, you could also just let it crash by
doing something akin to true = function(). Always go for let it crash first,
unless you really know you can handle the error sensibly (and then use a case-
statement)

3\. httpc is kind of known to not be the most formidable HTTP client. Hackney
is a more modern and better choice. The good thing with httpc is that it is
included with OTP, otherwise there are better choices (scalability- and
feature-wise)

4\. When someone says "multicore support" and "performance" support is not
great it's usually one of two things: (a) they're developing a use case that
is not fit for Erlang (e.g. compute heavy) or (b) they don't know how to use
Erlang properly (e.g. too complex process setups)

Now, many of these things are not obvious to people new to Erlang and takes
some experience to know about. This I would say is the bigger problem with the
Erlang eco-system.

~~~
ihuk
Agreed. Couple of clarifications/observations.

(1) there are situations like this:

    
    
        if
        DeductCreditInRatingGroup ->
          log("deducting credit...");
        end;
    

unfortunately you cannot write it this way. You have to write it as:

    
    
        if
        DeductCreditInRatingGroup ->
          log("deducting credit...");
        true -> ok
        end;
    

Not a huge deal but it's annoying.

(2) Let it crash attitude is something I never understood. It's just not
something you can do most of the time. If subscriber consumed 2MB and you let
it crash in the middle of rating function you just "lost" 2MB. With couple of
million subscribers on LTE that turns into very expensive error handling very
fast.

(3) We actually ended up using hackney.

(4) We were not doing anything compute heavy. I think our application was well
architected and we had no problem fitting it into OTP "framework". Comparing
CPU and memory usage to our Java implementation, Java is more performant.

edit: fixed code formatting.

~~~
e_proxus
1\. Here I would do something like this:

    
    
      log("deducting credit...", DeductCreditInRatingGroup),
    
      ...
    
      log(Msg, true) -> do_the_logging();
      log(Msg, false) -> ok.
    

Nowadays though, you should really use the new logger introduced in Erlang 21.
There you can do run-time filtering on many different parameters, or even
write your own handlers. Using if-statements is usually considered an anti-
pattern in Erlang, and I think very common coming from languages where only
if-statements exist. Problems are almost always better solved by using case,
or even better, pattern matching in function heads.

2\. Sounds like one of the rare cases where you care about the error and can
also handle it. In the use case you mentioned, I would buffer the data outside
of the loop that produces it and wrap the loop in a try-catch statement. That
way, you don't loose the data if one iteration crashes.

4\. Erlang usually ends up being good at latency and concurrency (scalability
over cores), together with a smaller and easier to read code base. I'm curious
about your case, and what would have made Java a better fit. If you share a
lot of global objects, Java might be faster for some things (at the cost of
concurrency usually). Erlang has tools for that as well though (e.g. ETS and
recently persistent_term).

~~~
ihuk
(1) was not about logging. I was trying to illustrate a valid use case for
`if` statement that requires `true` guard expression. We used lager[1] for
logging. I understand that new logger was inspired or loosely based on lager.

(2) I would argue that it's a rare case you don't care about errors.

(4) Multicore support in Erlang was not always great and you would run into
all kinds of problems after 2-4 cores. Than for a while it was 12-16 cores.
Not sure what the limit is these days.

Again, I am not arguing Java over Erlang. There is no lack of great examples
of large successful project running on Erlang. For me and my team Erlang was
just too much hassle for not much gain. Your milage may vary.

[1][https://github.com/erlang-lager/lager](https://github.com/erlang-
lager/lager)

~~~
lostcolony
Can't comment on your other issues (I'll say I have noticed the complete
opposite in practice, where the same team working with Java vs Erlang led to
far more reliable, performant systems in Erlang, but I obviously can't speak
to your experience), but for (2) -

You always care about exceptions/errors. The thing is, most you won't
predict/can't handle. Those you do predict, and know how to handle, you
should, because they're not really errors/exceptions at that point; they're
just edge cases.

The point of Erlang is not to make it so you just throw instead of addressing
an edge case; it's to make it so you reason about how, if something you don't
predict and don't know how to handle happens, you can get back into a known
good state. It's actually phenomenal at doing that. I've had complicated user
facing production systems work without noticeable issue for years (even while
under active development) in Erlang. I've never seen that in any other
language. Not to say it can't happen, I just haven't seen it, and the
operational lift to achieve that was no different for us than just "make it
work"; we were spending the same amount of time thinking about failure cases
with the Erlang system as other languages. The difference was a better
approach to handling the things we didn't think of.

------
jhgg
Although not in Erlang, Discord uses Elixir which runs on the BEAM VM to much
success as well for our real time distributed system as well as VOIP signaling
components. Looking back years later - it’s safe to say we wouldn’t have
chosen anything else.

~~~
erk__
What was the reason to choose Elixir over Erlang? Just familiarity with the
language?

~~~
superhawk610
Not OP, but have used Elixir pretty extensively; the major selling points for
me are the friendlier syntax and more active community support, plus with
rebar3 + Elixir's seamless Erlang interop you really don't give up anything,
you can use any existing Erlang/Elixir libraries with a single syntax. It's
pretty great.

~~~
alskdj21
Haven't touched Erlang but has an entry level knowledge on Elixir. I often
heard how easy it is to write macros on Elixir. Can you elaborate on this?

~~~
arjan_sch
One of the big differences between Erlang and Elixir is indeed that Elixir
allows you to write macros in the language itself, using quote / unquote
constructs just like lisps.

This makes it easy to generate code on compile time, which is then executed in
runtime without any performance penalty.

A large part of Elixir itself is actually implemented as macros. For instance,
the "unless" construct:

    
    
      defmacro unless(condition, do: do_clause, else: else_clause) do
        quote do
          if(unquote(condition), do: unquote(else_clause), else: unquote(do_clause))
        end
      end
    
    

(code simplified for clarity)

~~~
fcbrooklyn
And the if-else construct is also a macro. The whole thing ends up being a
case I think.

------
dominicl
Having worked extensively with both Erlang and Elixir in the last 7 years, the
main advantage for me has always been the smaller code base that those
projects tend to produce when compared to other languages. Especially for
projects that have some long-running components to them the Erlang built-in
features allow writing really short yet fault resistant systems.

The Supervisor system, the per process garbage collection (no stop the world),
'start_link', "Let it crash", and pattern matching are for me the superpowers
of beam. Elixir is contributing a great package ecosystem, 'mix format', and
better features for the new live web of websockets

~~~
verttii
Interesting. Any input on erlang vs elixir debate? Which so you prefer as your
primary tool personally?

~~~
fouc
You can easily drop into erlang from elixir, so you can have you cake and eat
it too.

~~~
sntran
Unfortunately the opposite way is not possible,yet.

------
theamk
At least some of those examples are highly misleading:

> There is no better example of Erlang’s reliability than the English National
> Health Service (NHS). [...] Using Riak (written in Erlang), the NHS has
> managed 99.999% availability for over five years.

From what I could find out, NHS's Spine2 is mostly Python, with "a bit" of
Erlang and Javascript. They do use Riak, but they also use Redis, RabbitMQ,
Python/Tornado, Ubuntu and lots of other components.

This sentence could have said "Using redis (written in C)", or "Using Tornado
(written in Python)", and still be fully correct.

~~~
ailideex
> the NHS has managed 99.999% availability

Really? I am not sure I trust this claim.

~~~
masleeds
The claim relates specifically to the spine 2 project, and is as measured over
the 5 years since it went live. There are a lot of systems which depend on
Spine 2 for their own availability, it is not the sort of thing that can go
down at any time without a lot of people noticing.

I'm biased in that I've been heavily involved in the project, but I will stand
by the five years of five nines availability claim.

------
atupis
I have never programmed Erlang but it feels like it is the currently only
language that is some kind of secret weapon. It has similar aura than Lisp had
before that with Erlang you can do stuff beyond "normal" languages.

~~~
ailideex
> It has similar aura than Lisp had before that with Erlang you can do stuff
> beyond "normal" languages.

Like what ?

~~~
rapsey
Erlang's advantage is being able to build a featureful, secure, scalable and
performant enough communication backend in very little time and effort. This
is a niche in which it can not be beaten.

~~~
otabdeveloper4
> ...in very little time and effort.

You have a very naive view of large-scale distributed systems. For companies
that truly need them, "time" and "effort" are never a consideration.
Furthermore, the challenges facing these companies in battling project delays
and cost overruns are the 100% organizational and political, _not_ technical.

~~~
rapsey
Where did I mention "large-scale distributed systems"?

My entire intention was to say Erlang is easy to get off the ground with
quickly.

Most companies outside of the Silicon valley bubble are resource limited.

~~~
otabdeveloper4
Let me unpack my way comment:

a) Erlang (as opposed to something else) is only useful for large distributed
systems.

b) If your organization is at the scale where you need a large distributed
system, then the problems in your project aren't related to code or to coding
speed.

~~~
rapsey
a) But it is not. Erlang is a very productive language and works great even
for small projects.

b) Needing a large distributed system is a requirement that one can get to
from lots of situations. You are making way too large of a generalization.

~~~
otabdeveloper4
a) _Every_ programming language is a "very productive language" that "works
great". Nobody ever thought to themselves "let's make a programming language
that's hard to use and wastes time". (And even if they did, they would have
never convinced other people to use it.)

b) No. Large distributed systems is not something you stumble onto, it's a
result of many years of organic growth.

------
MetalMASK
FWIW, wechat's messaging server is in erlang. In terms of scale, it's probably
bigger than whatsapp.

~~~
mcqueenjordan
The opposite is true by a non trivial margin. (It doesn’t change the fact that
WeChat is massive scale.)

[https://www.statista.com/statistics/258749/most-popular-
glob...](https://www.statista.com/statistics/258749/most-popular-global-
mobile-messenger-apps/)

~~~
mappu
The Discord number is inflated - 250mm is registered users, not MAUs. When
they announced 250mm registered, they had 56mm MAUs.

Telegram's 200mm is MAUs, and that number is over 1 year older than Discord's
information.

------
aloukissas
Awesome write-up, Francesco! To add to the success stories, our team's
previous startup (Bugsense) was a huge user of Erlang, which allowed to scale
to hundreds of thousands of concurrent devices with just a handful of servers
[1]. It was a no brainer that we would now use Elixir for our current startup
(AgentRisk [2,3]) and we wouldn't have been happier with our choice,
especially paired with Phoenix, which is hands-down the best web framework
we've ever worked with.

[1] [http://www.erlang-
factory.com/static/upload/media/1394234889...](http://www.erlang-
factory.com/static/upload/media/1394234889981861bugsenseldbefsf2014.pdf)

[2] [https://agentrisk.com](https://agentrisk.com)

[3] [https://blog.agentrisk.com](https://blog.agentrisk.com)

~~~
interactivecode
What would be a good resource to get started with Phoenix / Elixir, coming
from Node/EJS/React it seems very daunting

------
kitplummer
Fairly blind question: does Erlang|Elixir and BEAM make any sense in a pseudo-
embedded, edge compute or IoT environment with constrained resources (CPU and
memory), or even ARM architecture? Aside from the developer-side things, if
AMQP were to the centralized communications hub (e.g. RabbitMQ) does using
Nerves on devices make any performance benefit - meaning BEAM and app(s)
there?

The intent would be pre-process data and stream inward, or simply be a
command-and-control interface -> process and provide response.

~~~
Globz
Yes, take a look at nerves : [https://nerves-project.org/](https://nerves-
project.org/)

~~~
kitplummer
Yep, know Nerves. Like the toolset and the model. Really asking about the
underlying Erlang/Elixir that Nerves provides. The delivery flow (think image
to SD card) fits the more tradition embedded/realtime model - versus OS, and
installing app.

~~~
Globz
Well a Nerves image is a Unix Kernel + Erlang/Elixir & BEAM and whatever else
you need, so its a completely stripped down image. It uses the Whitelist model
so you add what you need instead of removing what you don't need. This is both
more secure and more performant since you don't have 100's of apps/services
running in the background, you only run what you need.

You can control everything remotely with nerves-hub, push update, debug etc.
Remember you only have to burn the image once, then its all update from here.

You can build a poncho application where everything is side-by-side, lets say
you have your app-firmware (you build image from this), then you build
separate app-ui, app-logic, app-whatever etc. and simply hard code the path as
dependencies in app-firmware and each app has their own configuration.

Everything will be bundled into a single image and protects you from things
leaking in from the sides rather than from above.

------
vasilakisfil
worked with Erlang lately in a side project, I think the actor model is really
cool (and the FP style), but the error messages are not helpful at all (at
least for a beginner), I wonder how other people manage.

~~~
fyfy18
Elixir gives you much more friendly error messages at runtime (I believe part
of this feature was contributed to the Erlang VM, so improved it for all
languages):

    
    
      10:21:21.525 mid=1 [error] GenServer {Registry.WatcherManagerRegistry, 1} terminating
      ** (CaseClauseError) no case clause matching: {:ok, <snip>}
          (core) lib/core/http.ex:64: Core.Http.request/7
          (core) lib/core/monitors/built_in_monitor.ex:99: Core.Monitors.BuiltInMonitor.perform_request/1
          (core) lib/core/monitors/built_in_monitor.ex:76: Core.Monitors.BuiltInMonitor.perform/1
          (core) lib/core/watcher.ex:40: Core.Watcher.perform_check/1
          (core) lib/core/watcher_server.ex:91: Core.WatcherServer.handle_info/2

------
tickbw
I have been building grouper.ai to integrate voice ai systems with 2600hz's
kazoo apis, and have enjoyed doing it as a Greenfield side project. Nerves for
hardware on embedded Linux, c/c++ for the voice apis, ports in erlang/elixir
to talk the low level calls, using elixir Phoenix in the cloud on the beam.
Have used erlang also in day job for custom industrial M2M protocol
translation (serial / Ethernet) very successfully also.

------
atoav
I have seen that Erlang is used in RabbitMQ, if I recall right their argument
was that Erlang is particularily well suited for that kind of asynchonous
message queue stuff. I donlt know all too much about Erlang, so I cannot judge
if that claim makes sense.

~~~
zbentley
I'm an Erlang novice, so take this with a grain of salt: Erlang's use in RMQ
seems to have helped them immensely through the first few years of the
system's life. They got an "all in one box" performant message broker off the
ground, and relatively reliable, quite quickly.

However, RMQ did hit numerous issues that they had to work around on the
Erlang platform. They had to engineer what was effectively their own scheduler
on top of parts of the native Erlang one in order to prevent some starvation
cases [1], and I have also heard that issues with Mnesia are at the root of
some of the pathological behaviors (data loss) of some versions of RabbitMQ
when restarting from a crash (this is admittedly anecdotal, so take it with a
grain of salt).

[1] [http://alvaro-videla.com/2013/09/rabbitmq-internals-
credit-f...](http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-
for-erlang-processes.html)

------
HeXetic
My only exposure to Erlang has been through RabbitMQ and while I can't comment
much on the language itself, managing the packages is a dependency nightmare.
Like some twisted, mutant, supervillain version of Python, a given version of
RabbitMQ will only work with a certain version of Erlang (with horrible, hard-
to-understand error messages on incorrect versions), and on RedHat/CentOS, the
Erlang packages all misadvertise what exact versions they supply,
necessitating awkward forcing of the dependencies or else Yum screws
everything up.

~~~
liveoneggs
why wouldn't you just use rabbit rpms that depend on the exact correct
versions of erlang? aka els-erlang

[https://www.erlang-solutions.com/resources/download.html](https://www.erlang-
solutions.com/resources/download.html)

Linux distros have always been terrible places to find reliable packages.

~~~
oblio
He he. And then you have people complaining that you're a "bad OSS citizen"...

~~~
liveoneggs
me, ELS, or redhat?

------
miki123211
> Many countries rely on Erlang for their immediate payment switches, allowing
> instant bank transfers

Just wondering if such a system used in Poland, actually called Elixir[1], has
a line of Erlang/Elixir in its codebase.

[1]
[https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międz...](https://pl.wikipedia.org/wiki/Elixir_\(system_rozliczeń_międzybankowych\))

------
mc_
We're happily plugging along on KAZOO [0] since 2010 at 2600Hz. Erlang is a
force-multiplier for us. Approaching 300K LOC and ~10 Erlang engineers
(depends if Karl is writing code or CTO-ing that day) today.

[0] [https://github.com/2600hz/kazoo/](https://github.com/2600hz/kazoo/)

------
latentpot
IIRC a clearing co. CLS used erlang a few years ago. LinkedIn still has the
erlang architect JDs somewhere.

------
z3t4
Something I find interesting is that the tech stack is often chosen by non-
engineers/founders. So languages and frameworks are marketed to business
people. Freelance work is often weak on the requirements, but they have strong
opinions on what language and framework to use.

~~~
lagadu
That's often because most clients will already have some sort of environment
set up, running some OS and with certain features installed and most
importantly: they don't want to have 5 different projects working on 5
different frameworks because that'd be a nightmare to maintain. Once a company
gets into a tech stack, they'll want to stay within it as much as possible and
that makes plenty of sense.

~~~
z3t4
You dont want to put all your eggs in the same basket, not in this age when a
framework can become obsolete in just two years. It only make sense if you
already have a small team - you want to use what they know best. But if you
are growing you can have many small teams. You will also exhaust the talent
supply if you lock yourself down to only one platform/language/framework.

~~~
oblio
Yeah, tell that to Java/Spring or C#/ASP :-)

------
aloer
What would be a good use case to learn erlang/elixir with that is not just a
chat app?

Online gaming?

Collaborative tools?

~~~
kerkeslager
A Bittorrent client is a pretty good one. The protocol is fairly well-
documented, and the domain is very well-suited to the language.

~~~
vonseel
I'd use this if someone made a good one! I'm currently using Transmission on
Mac, which has more than its own fair share of issues, most notably the UI
freezes in any action that takes time, like moving a file across hard disks or
to a network drive, but IIRC all the others I have tried (Deluge, qBittorrent)
also have multiple similar issues.

------
truth_seeker
Is there any official source which talks about the number of servers or total
CPU cores used in production by the companies mentioned in the article?

------
fastbmk
Observed facts of Google Trends:

1\. Erlang/Elixir are less than 1% of programming languages market.

2\. Erlang/Elixir popularity is trending down.

~~~
therockhead
Elixirforums have reported year on years increased traffic, so I’m surprised
to hear that.

------
known
Telecom companies

~~~
melicerte
So you think gaming, online bets, financial institutions or NHS are telecom
companies?

~~~
hsson
I'm fairly certain he/she refers to Ericsson, the creator of Erlang.

------
wprapido
teamweek.com uses Elixir

------
aloknnikhil
WhatsApp used to use Erlang, not anymore. Facebook rewrote it completely in
C++.

[https://www.quora.com/When-did-Facebook-switch-away-from-
usi...](https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-
for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with)

~~~
bdd
This isn't true at all.

Major functions of WhatsApp are implemented in Erlang and they're actively
worked on.

Source: Self. Engineer at FB.

~~~
zerr
They (you) had to modify beam vm heavily though.

~~~
rapsey
At that scale you are generally modifying your entire stack or building it
from scratch no mater what you are using. Also they were on a pretty old
version and a number of their fixes are in Erlang itself now.

WhatsApp engineers have been quite clear that using Erlang was a great choice.

