
Why Erlang Matters - _oakland
https://sameroom.io/blog/why-erlang-matters
======
atemerev
AFAIK, Erlang is still (as of 2016) the only distributed actor model
implementation with preemptive scheduler.

All other major implementations (including Akka) have cooperative scheduling,
i.e. forbidding blocking code in actors. Erlang allows it. This is huge.

And actor supervision is the best way to write reliable systems. I have wrote
some code in Akka without much effort and testing (streaming market data
aggregation), and it is still running with a few years uptime.

~~~
incepted
> And actor supervision is the best way to write reliable systems.

Need some citation there, because we've been writing extremely reliable
systems (multiple nines) for decades in various languages (mostly C, C++ and
Java).

All these languages (including Erlang) come with pros and cons to write such
systems, but so far, Erlang has failed to deliver on most of the promises that
its advocates repeatedly make.

~~~
daveguy
> but so far, Erlang has failed to deliver on most of the promises that its
> advocates repeatedly make.

Speaking of citations...

~~~
incepted
Well, Erlang is a niche programming language, isn't that evidence enough that
all these claims that Erlang will save us from the multithreaded hell are
vastly overblown?

We're doing multithreaded programming just fine in a lot of very varied
languages.

------
jandrese
I'm a very green Erlang noob, but given what I have seen from it I find
articles like this kind of strange. Sure concurrent programming is difficult
and we need to think hard about how to make programs run quickly in a
multiprocessor environment, but the fundamental architecture of Erlang seems
to be in conflict with big data and high speed computing. It seems like a
language that can scale much better, but has such enormous constant time
penalties that the scaling can't overcome the hurdle until you're talking
about thousands of processors.

Every single state change requires a function call, every function call
involves a pattern matching algorithm that has to account for all of your
program arguments (effectively almost the entire state of your thread!) and
isn't even strongly typed. And then there is synchronization and data sharing
between threads, which seems a bit handwavy and effectively requires a
database running in your program that the threads can poll.

If your problem set is lots and lots of small independent tasks that don't
have to finish overly quickly, then Erlang is fantastic. Stuff like switching
voice circuits for example. But I'm trying to imagine manipulating a 1TB
dataset with thousands of worker threads if you have to make a copy on the
stack for every change every worker makes.

I know people do some big data stuff with Erlang, so these have to be solved
problems somehow, but I can't help but to suspect that they have to compromise
some of the ideals espoused in this article to make it work.

~~~
jerf
Erlang may be useful for coordinating computation tasks, but, yes, even with
HIPE it is not a good numerical language on its own.

It would be interest to re-engineer a language today that tries to fit into
Erlang's niche but has a stronger performance focus. Rust, Cloud Haskell, and
Go all sort of cluster in the area I'm thinking about, but none are quite what
I'm thinking of. Cloud Haskell is probably closest but writing high-performing
Haskell can be harder than you'd like. Rust shows you don't have to clone
Erlang's immutability and all the associated issues it brings with it for
inter-process safety, but Rust of course is "just" a standard programming
language next to what Erlang brings for multi-node communication.

~~~
wcummings
You could write NIFs in Rust (well not sure if you can now, but I don't see
any reason it couldn't be supported) for the high perf bits and use Erlang to
coordinate, I figure. At least Rust code is less likely to explode and bring
down the whole VM than C.

~~~
arh68
Any idea if a network-level FFI has been started? I'm thinking along the lines
of the Haskell erlang-ffi [0].

    
    
        Speaks the Erlang network protocol and impersonates 
        an Erlang node on the network. Fully capable of 
        bi-directional communication with Erlang.
    

NIFs still limit you to < ~1ms computations, from what I understand, but
impersonating a node (on another machine, even) seems a lot more flexible.
Just wondering; NIFs in Rust are still a great idea.

[0]
[https://hackage.haskell.org/package/erlang](https://hackage.haskell.org/package/erlang)

~~~
toast0
Have you seen c nodes[1]? That may be what you're looking for?

[1]
[http://erlang.org/doc/tutorial/cnode.html](http://erlang.org/doc/tutorial/cnode.html)

------
rb808
I'm really tempted to use Erlang/Elixir for a project at work but unsure of
its traction. Is it leading edge or trailing edge? I don't even know, but
don't want to saddle the firm with a white elephant - even one that is
impeccably fault-tolerant.

Is Erlang too esoteric?

~~~
dmix
Having spent quite a bit of time searching for Erlang packages on Github
recently, I've found quite a few Erlang packages haven't had commits since
around 2013. I believe there was a spike in adoption around 2012-2013 which
resulted in a bunch of activity on Github. Which seemed to have tapered off
recently. But you still find many core packages that are active - things like
JSON parsers, templating libraries, and web frameworks are all lively. And OTP
and Erlang itself are always being improved.

Plus there is an up-to-date package manager/global repo
([https://hex.pm/](https://hex.pm/)) which is used by both Erlang and Elixir.
Plus Rebar3, the primary build tool for Erlang projects, is actively
developed.

But Elixir packages are very active in general since it's relatively new and
these packages can be used within Erlang apps relatively easily as well. Plus
the Erlang packages from 2013 that I've used have all been pretty stable. The
quality of libraries available on Github have all been very high from my
experience. I believe Erlang attracts experienced developers, which is
reflected in the typical code quality and documentation.

My only wish is for the Erlang website
([http://www.erlang.org/](http://www.erlang.org/)) to get a redesign. It gives
the language an esoteric apperance to newbies. Which is a shame because I
absolutely love the language and think it's far superior for many of the types
of projects for which people have been using Node.

~~~
rdtsc
I am excited for a continued improvement and enhancement to the language.

Erlang 18 brought maps, people complained about that for years. Erlang 18 also
brought (via a feature flag) dirty schedulers so can have long running C
function embedded in without messing up process scheduling also an often
requested features. Probably the best thought-out handling of time in any
language I've seen so far ( synchronization, warping, moving backwards:
[http://erlang.org/doc/apps/erts/time_correction.html](http://erlang.org/doc/apps/erts/time_correction.html)
)

Erlang 19 is exciting as well -- 10x faster tracing, dirty schedulers turned
on by default, a new state machine OTP module, an external plugin (with
LevelDB as one example) for Mnesia storage also something people complained,
2x-3x faster spawning external processes + many others.

The impressive part is that these changes are done to a 30 year old language.

~~~
simoncion
mnesia_leveldb is scheduled for OTP 19? That's great news!

I can't tell you how many times I've cursed at the overfilled-hours-ago-but-
mnesia-didnt-care DETS table shard. Getting a on-disk backend that can store
more than 2GB at a time will be great!

It'll be even better if the writer code actually _notices_ failures to write
to the backing store and aborts transactions when they happen! [0] :)

[0] Seriously, who thought it was a good idea to ignore the return value from
dets:insert/2? :(

~~~
rdtsc
I think this is the PR so far:

[https://github.com/erlang/otp/pull/858](https://github.com/erlang/otp/pull/858)

------
amsha
Any programming language can do IPC, but Erlang/Elixir is the only language
I've seen that makes it really easy. Erlang IPC is transparent whether you're
sending messages between to local processes or remote processes. Most
languages have trouble with local IPC (usually because of race conditions in
data). Every other language I know of, including Go, needs custom code to
handle remote IPC.

~~~
MichaelGG
It just seems that all the effort going into Erlang would be better spent on
making the OTP stuff as a library for a popular, cross-language platform. Like
the JVM (or maybe .NET.)

~~~
querulous
sure. we can start when the jvm can do preemptive scheduling of processes

erlang isn't great because of a single feature or library, it's great because
it's designed from the ground up for one very particular role. you can't just
port the otp api to go or the jvm. you need the whole foundation

------
waf
I'm really interested in BEAM languages, but the fault-tolerance / supervisor
aspect of it doesn't speak to me. Aren't all modern application fault-
tolerant, as long as you don't design something really poorly?

For example, I've never had a single HTTP request bring down an entire website
-- that's already isolated. Same with message-queue listening processes. For
general batch applications, I've always had them short-lived and running
periodically, e.g. every minute, so even a complete crash there is isolated
between runs.

One powerful aspect is how it strongly encourages you to design loosely-
coupled message-passing systems that should be easier to scale out. But I'm
not convinced that's enough to warrant a switch.

~~~
ngrilly
Erlang's fault-tolerance becomes really useful when you wrote a server that
manage hundreds of thousands of simultaneous connections (a chat server being
the typical example). With Erlang, each connection is managed by its own
lightweight process (no callbacks, no promises, etc.). If a lightweight
process fails, it doesn't bring down the other processes. Moreover, BEAM can
signal other processes about the failed process (Erlang' supervision trees are
based on this mechanism).

In a traditional architecture, you would use one thread for each connection
(let's ignore the issue of the memory used by each thread), but when one
thread fails, it would bring down all connections instead of just the failing
one.

------
tombert
You know, even if Erlang didn't have great SMP scaling, or wonderful
distributed properties, I think its fault-tolerant, actor-model-ey nature
would make it wonderful anyway.

The fact that you program expecting failures, and forced isolation of
everything allows for incredibly "sturdy" code. The other features are
fantastic, but they're just gravy as far as I'm concerned.

------
atemerev
The future is already here: [http://erlangonxen.org/](http://erlangonxen.org/)

~~~
thenewwazoo
Ling is cool stuff, but I have the impression that it's a clean-sheet Erlang
VM implementation, not a port of BEAM. BEAM is time-tested and battle-proven,
and I'd really like to see BEAM itself rely less on the underlying OS (e.g.
epmd as a separate OS process, quirks in how it uses select/poll). I know Peer
Strizinger did a lot of work on this[0], but hasn't (to my knowledge) yet
released any of his work, sadly.

[0] [http://www.grisp.org](http://www.grisp.org)

------
bitmadness
Erlang is glacially slow. Even on a 20 core machine, a multithreaded Erlang
implementation will usually be trounced by a good singlethreaded C++/Go/Java
implementation. All this stuff about multicore scaling is baloney - who cares
it is scales and is still slow?

~~~
rdtsc
> multithreaded Erlang implementation will usually be trounced by a good
> singlethreaded C++/Go/Java implementation

And Go/Java implementation can be trounced by hand written assembly and ASIC
accelerators probably.

> All this stuff about multicore scaling is baloney

You say baloney I say money in the pocket. I've seen it scale, I've seen it
work reliably in large clusters, I've been able to inspect, debug and hotpatch
running systems while they are still running. I have seen systems which had
non-critical components crash and auto-restart for days without impacting
customers and needed teams of "devops" to babysit it.

Moreover I've see single and multi-threaded C++ and Java applications with
threads and and data races which take weeks or months to find. Or they are
screaming fast until they take a nosedive and segfault (also in some minor
stupid new feature which nobody uses). You know what the transaction
processing rate of a segfaulted process is? - 0 tps.

That's why teams like Whatsapp could get by with only 10 or so back-end
engineers handling billions of messages / day from various devices new and
old, while other companies need 10x or even 20x more than that.

It is not just being able to run fast. Assembly runs very fast. It is also
about being able to have the right tools and abstraction to define a problem.
Erlang has those and they come built-in (the OTP library, the distribution
protocol etc), C++ doesn't, so have to start from STL and boost and so on,
then get serialization, monitoring, supervision, etc bootstrapped.

------
eddd
To me the power of Erlang comes from a combination of powerful VM and OTP. In
the era of hype for SOA, OTP

You can run multiple apps on the same VM and they will run concurrently and
communicate with each other using protocols that are core of the language.

I agree, erlang seems weird at first, simply because there is not anything
like it. It also solved todays problems with software a decade ago.

------
d33
One word: ejabberd. That's why erlang matters to me :)

~~~
d33
Also, it has a great free book on it:
[http://learnyousomeerlang.com/](http://learnyousomeerlang.com/)

~~~
simoncion
There are at least two great free books: [http://www.erlang-in-
anger.com/](http://www.erlang-in-anger.com/) ! :)

~~~
rvirding
Written by the same author no less. :-)

------
ssmoot
This seems like it could've been called "Why Actor Systems Matter".

If you're on the JVM, I'm not sure what Erlang buys you in practice. It's
slower and more obscure. It has a much smaller ecosystem. While process-safety
is frequently touted, in the real world this is a non-issue among non-issues.
It's just not an actual thing. It's not like the JVM goes around Segfaulting
all the time.

I'm totally sold on Actor Systems and think it's something more programmers
should expose themselves to. I'm just not sure there's much of an argument for
Erlang vs the JVM unless you're completely sold on the notion of process
isolation for some reason.

~~~
Xixi
It's not at all about segfaults. Assuming the Erlang VM is as likely/unlikely
to crash as the JVM, then it's a complete wash: Erlang processes are green-
threads, not OS processes, so you don't gain anything there.

It's about the share-nothing architecture of Erlang: it means your processes
are isolated and can be stopped/restarted independently, crash/hang without
writing into the memory of another process, have their own garbage collection,
and be moved to a different physical computer (or even data center) with no
impact at the logical level. Erlang forces you to architecture your program as
a collection of nano-services. You can of course emulate some of it on the
JVM, but you can't go lazy and cut corners with Erlang.

~~~
ssmoot
Akka checks most of those boxes AFAIK. And there's really not much you can do
in the way of cheating unless I misunderstand you. Or at least idiomatically
you wouldn't cheat in Scala anyways.

Speaking of which, I just realized the AtomicLong I'm using in my
IdGenerationActor (performs an atomic increment of a processId counter in the
database, then uses that with the AtomicLong as the input to Hashids; fast,
in-process, cluster-safe short-Id generation that will leave popular Redis
based solutions in the dust) is completely unnecessary. Copied and pasted from
non-Actor code without consideration.

I guess Actors still can't keep you from doing dumb things yet. ;-)

~~~
querulous
this is just not true. every other week i have production issues because akka
managed to exhaust the thread pool with long running/nonresponsive actors

~~~
ssmoot
You've done something wrong then? Or maybe using experimental features? I
might play with them a bit, and it can be frustrating to wait, but I've never
launched anything on -experimental before.

I was referring to "cheating", with the idea that you might pollute your
Actors with... I dunno. Programmatic connection pooling for your database
driver? Passing mutable messages around?

Both of those things would be very unusual in Actors written in Scala since
pretty much everything is immutable by default. Seeing that sort of thing
should at least raise some eyebrows.

As far as non-responsive, I've never seen that. But I do take care to use
Futures where appropriate, and pipeTo. Maybe it's good habits, or maybe I'm
just very lucky.

One of my first Akka projects is still running, still processing content it's
notified of by Postgres through LISTEN/NOTIFY, still posting that content into
Cloudant (basically a managed Lucene deployment in this case).

And it's been running since September 2014 without a restart AFAIK. Which
would have never happened with a previous non-Actor solution we might have
used if for no other reason that a network blip might detach the listener,
whereas here the Actor just restarts.

My experience has been overwhelmingly positive. Despite doing the wrong thing
occasionally.

If you have Actors that are hung, I guess the first thing to try is figuring
out which one(s). After that, you could just schedule the supervisor to
routinely PoisonPill them, and forcefully stop them after a grace period.

I'm not sure that OTP is going to help with this sort of issue either. You
have an apparently blocking process that makes an Actor non-responsive. The
fact that it's single threaded within the Actor and stalls it's mailbox is
kind of the point of Actor systems AFAIK.

I guess one thing I feel like helps me is to keep my Actors small and doing
one thing. It's hard to do too much damage when you only have a dozen lines of
actual message handling. "One thing" is sometimes coordination/aggregation
BTW. Which means I might have a GetRequestActor, PutRequestActor,
DeleteRequestActor, etc. Which do the one thing. But then I also have a
DatabaseActor that basically just coordinates/forwards for all those so you
don't actually have to deal with them yourself.

And from there if I decided that no GetRequestActor should take longer than 10
seconds to do it's thing, I can easily schedule the DatabaseActor to
forcefully stop any task exceeding it (without the need for an actual watch if
you choose). Which you can then log an ERROR for and work out why. And maybe
some requests just take longer and that's OK. So maybe you then write a
LongRunningRequestPathExtractor and put that pattern before the normal one.
And now you can have multiple timeouts for different paths.

Or maybe you just record the epoch for different requests, and if you've had
1,000 updates since the last View request, you know the next one is going to
trigger a reindex. So you give it extra time. Or you set it to allow stale
results and reschedule the same call to occur again in 1 minute to minimize
the disruption of indexing the changes. Just some thoughts.

~~~
coldtea
> _You 've done something wrong then?_

Well, the idea is with Erlang you can't.

~~~
ssmoot
I don't believe Erlang goes around magically imposing it's own timeouts on
message handling.

~~~
querulous
it doesn't, but an erlang process that is waiting on a message won't block
other processes from running ever

------
simula67
Please let me ask some noob questions.

> Erlang matters today because it demonstrates how these semantics can be
> elegantly packaged in one language, execution model, and virtual machine.

Why is it so important to demonstrate that these semantics can be packaged
into such a homogeneous environment ? Is it even a good idea ? Is this way of
doing things superior to having micro-services that talk to each, all managed
by a supervisor system ? Wouldn't that allow us to take advantage of the
unique upsides of multiple programming languages, virtual machines and
execution models ?

~~~
Tomte
You may underestimate the difficulty of writing a bullet-proof and feature-
rich "supervisor system". That has pretty high complexity and lots of nasty
edge-cases.

Erlang happens to nail that part with OTP.

------
zwischenzug
I made an argument that we were reinventing many of the ideas of Erlang in the
DCOS model (Docker, Kub, and cloud etc).

Text is here (CTRL-F erlang):

[https://zwischenzugs.wordpress.com/2015/11/17/dockerconeu-20...](https://zwischenzugs.wordpress.com/2015/11/17/dockerconeu-2015-talk-
you-know-more-than-you-think/)

Video here:

[https://www.youtube.com/watch?v=-qHwL8C9UoA](https://www.youtube.com/watch?v=-qHwL8C9UoA)

------
tmerrifi
"Cache coherence doesn't scale."

This is a controversial statement, and an opinion that is not shared by many
respected computer science researchers:
[http://research.cs.wisc.edu/multifacet/papers/tr2011-1_coher...](http://research.cs.wisc.edu/multifacet/papers/tr2011-1_coherence_stays.pdf)

~~~
brazeon
It does scale when being considered by software running on it. Even more with
some alternative approaches like directories. What OP means is that it does
not scale for software to be oblivious to underlying cache coherence and to
operate on "one flat shared RAM" assumption. E.g. false cache line sharing
etc.

------
ragnar123
Linked article "power-wall"
([http://daimi.au.dk/~zxr/papers/treewalls.pdf](http://daimi.au.dk/~zxr/papers/treewalls.pdf))
seems to be unavailable.

Does anyone working link to this article?

~~~
rekoros
Updated to point to [https://en.wikipedia.org/wiki/Multi-
core_processor#Technical...](https://en.wikipedia.org/wiki/Multi-
core_processor#Technical_factors)

------
dboreham
Doesn't the existence of Golang remove most of the reasons to use Erlang these
days?

~~~
jlouis
I was wondering this when Go came out. But time has shown they solve problems
very differently, so they are not really direct competitors.

Erlangs primary difference over Go is that it can gracefully handle the case
where a programmer has made an error in the program by accident. A typical Go
program cannot gracefully recover from an error which were unforseen and never
considered by the programmer. The Erlang equivalent Erlang program, however,
has mechanisms to safely clean up resources for the faulty part and then get
on solving work. This is the reason Erlang has a nice robustness story.

~~~
andy_ppp
[https://github.com/thejerf/suture](https://github.com/thejerf/suture)

Supervisors for golang. It looks excellent. I personally have fallen for
Elixir and think the syntax, immutability, community and class functionality
like Phoenix.Presence make me bet that it'll be bigger than Python for jobs in
5 years.

~~~
jlouis
Supervisors is only half the game here. Suppose A sends a message to B and
decides to wait around for the answer. B now divides by zero.

In Go, your whole program is in consistency trouble and you have to write code
to handle the case.

In Erlang, your monitor on B means you get told it is dead via an async
exception delivered into your mailbox.

I know what kind of system I want to work with here :)

~~~
andy_ppp
Wow. That's just incredible. I seem to _keep_ saying that the more I learn
about Erlang/Elixir.

------
amelius
One missing feature is that you cannot run the same code in the browser and on
the server (which is very useful).

Is there a language that compiles to both Erlang and Javascript?

------
dschiptsov
There is Armstrong's thesis (google it) which explained all the major design
decisions much nicer that this arrogant "Erlang matters".

The foundation principles is not only in selecting a functional language and
enforce immutability, but explicitly rejecting all sharing (threads in the
first place) and providing a theoretical basis of why JVM is an unacceptable
target for reliable, soft-realtime systems, (no matter what Scala guys might
tell you) - a crashed process would affect none other, no data corruption, no
locks, no messed up stack. This is what is behind the "let it crash" meme.

Erlang is not "matters", it is a masterpiece of software engineering to study
and learn insights from. Especially, how to make ones own decisions based on
right principles and rejecting sectarian dogmas (run everywhere!) of wast
majority.

