
What Erlang Taught Me About Distributed Systems - kenforthewin
https://blog.kenforthewin.com/what-erlang-taught-me-about-distributed-systems/
======
julienmarie
I discovered Erlang in the late 2000s via its kinda weird and iconic video
"Erlang the Movie"
[https://www.youtube.com/watch?v=xrIjfIjssLE](https://www.youtube.com/watch?v=xrIjfIjssLE)
that prompted me to dig further. I was at the time an average developer.
Erlang, and Joe Armstrong books and videos, made me understand things that
other web-oriented languages ( Ruby, PHP... ) never really dug into at the
time. Because the Erlang ecosystem is more than distribution. Distribution is
a consequence of its design around concurrency and message passing and let-it-
crash/auto-healing philosophy. Erlang, and after that Elixir, made me think
differently about code and made me a better programmer.

~~~
paulsutter
The sequel is shorter and more fun, “all erlang needs is an image upgrade.. we
need something fresh, something edgy”

[https://youtu.be/rRbY3TMUcgQ](https://youtu.be/rRbY3TMUcgQ)

~~~
cuddlybacon
I love it!

When will OTP be updated to support Blockchain?

~~~
kirillseva
Haha, this reminds me of red-lang's approach to get funded via ICO
[https://ico.red-lang.org/](https://ico.red-lang.org/)

------
outlog
Virding's First Rule of Programming:

 _Any sufficiently complicated concurrent program in another language contains
an ad hoc informally-specified bug-ridden slow implementation of half of
Erlang._

[https://rvirding.blogspot.com/2008/01/virdings-first-rule-
of...](https://rvirding.blogspot.com/2008/01/virdings-first-rule-of-
programming.html)

(he is one of the creators of erlang / stars in Erlang the Movie)

~~~
TurboHaskal
That's funny, considering that the reason I'm doing concurrent programming in
"another language" is due to Erlang's poor performance.

~~~
nickh9000
Erlang is not a panacea. It excels as a middle layer, one that orchestrates
work. It's not a tool to perform intensive calculations.

------
nudpiedo
I expected a bit more of content in the article, like which components of
kubernetes equals which OTP functions or general patterns and how easy can be
to implement your mini-kubernetes on Erlang, not just “hey they both solve the
problems of a distributed by using similar patterns”.

I wish I could downvote but this feature does not seem available in my UI, I
guess only certain users can downvote.

~~~
di4na
The main difference sadly break most of the analogy.

Because a pod is far from the granularity of a process. It is closer to the
granularity of an application, the biggest abstraction that OTP gives you.

What that mean is that k8s does not have the same property than erlang. In
erlang, isolation is your go to tool. If you need something, you spawn a
process. Having a lot of really granular process makes scheduling on cores
easy, makes your design easy to crash because the blast area is small, and
makes it really cheap to have message passing in the core of the VM.

In k8s, any networking is hard, hence the current trend of service mesh. As
the unit of computation is bigger, supervision get harder. Building a
supervision tree is more expensive. And crashing and restarting get more
expensive too.

K8s has the advantage of being language agnostic ofc. But do not forget the
paradigm shift that having really cheap process give you in an integrated
environment like the BEAM.

And ofc, i have not even talked about the dynamic tracing that the BEAM gives.

------
gtycomb
I recently went half way in a project using Erlang back-end but didn't get to
a production stage mainly due to political reasons in my organization. Mnesia
(Erlang’s distributed database) is remarkable and in the Erlang way it comes
completely packaged with the system. My only gripe is that documentation
overall is a bit hard once you start digging in. The OTP book mentioned is
good but may be a bit dated (but we can figure out what is amiss still). I am
not sure whether drivers for accessing external databases such as Mongo,
Postgresql etc are actively maintained. (Edit: when I check it now it looks
like the drivers are recently updated!)

~~~
pipu
Do you have any other book suggestions?

~~~
gtycomb
A nice application oriented book is the 'Handbook of Neuroevolution Through
Erlang' by Gene I. Sher, from Springer. It has the code and a model for
distributed agents (for neural networks in biology)

------
enraged_camel
>>Elixir depends on the decades of development behind the Erlang VM and in
many ways is just syntactic sugar on top of Erlang/OTP

I think calling Elixir "just syntactic sugar" is a gross and unfair
oversimplification. While friendly syntax is one of its welcoming qualities,
Elixir is its own language. It doesn't "transpile" into Erlang. It _compiles_
down to bytecode.

~~~
dmitriid
[https://medium.com/@fxn/how-does-elixir-compile-execute-
code...](https://medium.com/@fxn/how-does-elixir-compile-execute-
code-c1b36c9ec8cf)

Elixir creates an AST which is transformed into Erlang Abstract Format
([http://erlang.org/doc/apps/erts/absform.html](http://erlang.org/doc/apps/erts/absform.html))
which is then compiled into Erlang bytecode by Erlang compiler.

Erlang VM's bytecode is largely undocumented (except for third-party attempts
like [http://erlangonxen.org/more/beam](http://erlangonxen.org/more/beam) and
[http://beam-wisdoms.clau.se/en/latest/](http://beam-
wisdoms.clau.se/en/latest/)). So all languages on the Erlang VM either create
and compile Erlang Abstract Format or transpile to Core Erlang
([https://8thlight.com/blog/kofi-gumbs/2017/05/02/core-
erlang....](https://8thlight.com/blog/kofi-gumbs/2017/05/02/core-erlang.html))

~~~
gootik
Don't forget about
[https://github.com/happi/theBeamBook](https://github.com/happi/theBeamBook)

------
amelius
Does Erlang support structural sharing between large data structures? If not,
how is it possible to efficiently pass large messages between processes on the
same physical machine?

~~~
zbentley
Passing on a local machine is optimized quite well. The API is the same as
passing to a remote machine, but serialization is omitted when passing
locally. A fair amount of copying still happens, but (I assume--now we're into
territory that I am less sure about) that keeps the implementation robust and
simple (i.e. there's no "ownership" to wrangle down in the BEAM; at the
highest level of abstraction inside the VM, it's just copies of immutable
structures).

For communicating quickly locally, you could use large/refcounted binaries
([http://erlang.org/doc/efficiency_guide/binaryhandling.html](http://erlang.org/doc/efficiency_guide/binaryhandling.html))
in to avoid copying-on-message-pass. Those generally tend to be frowned upon,
and incur the usual headaches/perf issues of a ref-counted structure as well.

In short, I'd suggest defaulting to the assumption that Erlang is fast
_enough_ at passing messages locally for it not to matter that much in
practice. Benchmark the throughput of your system locally, and if the overhead
of message passing isn't a major pain point immediately, move on up the stack.
If you're doing scientific computing and want to number-crunch huge data sets
in parallel via some minimal-coordination memory-map-backed system, this is
not the platform for you (or use a NIF).

~~~
CyberDildonics
How is it optimized well when there is still lots of copying going on?
Anything in the same process space should just be a matter of passing a
pointer or a pointer with very little information wrapping it up.

> Those generally tend to be frowned upon, and incur the usual headaches/perf
> issues of a ref-counted structure as well.

Why would reference counting be worse than copying?

~~~
di4na
Because refcounting means lot of cache miss on the CPU core.

One of the interesting advantage of erlang is that its processes are small and
mostly fit in L2 memory. So you can load a whole process at once, then compute
for your whole slice, all in cache.

The result may surprise you.

~~~
CyberDildonics
Why would reference counting increase cache misses?

> One of the interesting advantage of erlang is that its processes are small
> and mostly fit in L2 memory.

Are you talking about data or are you talking about execution? I wouldn't
think execution would need reference counting.

> So you can load a whole process at once, then compute for your whole slice,
> all in cache.

Whatever data is being loaded still has latency. If it is accessed linearly it
can be prefetched. If it is being copied around, that won't won't put it into
a CPUs cache any faster and will require using writeback cache, not to mention
all the cycle of copying.

I'm not sure how data that already exists in memory and potentially in L3
caches, etc. is going to be hurt by reference counting.

> The result may surprise you.

Do you have results that I can look at? If not I think they will surprise both
of us.

------
ngrilly
> To wrap up, it's evident that much of the work that went into Erlang has
> inspired the work of Kubernetes.

Even if there are clear similarities between Erlang and Kubernetes, I'm not
sure the former inspired the latter. We should ask Kubernetes developers :-)

~~~
zaphar
Kubernetes is based on borg which as far as I know wasn't really inspired by
erlang necessarily. However I suspect the problem space lends itself to
idenpendent discovery of very similar solutions.

------
z3t4
With all this message passing, have anyone experienced the network becoming
the bottleneck !?

~~~
zbentley
In large Erlang applications, the "send to multiple peers" broadcast
tendencies of its message passing can indeed cause the network to become a
bottleneck; so can the default chattiness that a lot of Erlang application
developers assume for their app behavior.

Fortunately, there are lots of simple-to-apply patterns and solutions when you
get to that point, for example the excellent Partisan library:
[https://github.com/lasp-lang/partisan](https://github.com/lasp-lang/partisan)

------
ngrilly
Is it possible with Erlang to make sure that two Erlang processes are co-
located on the same physical node?

~~~
_asummers
Yes! Each node has a name and you would look into the functions on the Node
module on how to do that. In Elixir you'd say:

    
    
      iex(3)> defmodule Hello do
      ...(3)>   def world() do
      ...(3)>     IO.puts "here we go!"
      ...(3)>   end
      ...(3)> end
      iex(4)> Node.spawn_link(:erlang.node(), fn -> Hello.world() end)
      here we go!
      #PID<0.101.0>

~~~
ngrilly
Thanks! I should have read Erlang docs:
[http://erlang.org/doc/man/erlang.html#spawn-2](http://erlang.org/doc/man/erlang.html#spawn-2)

------
pipu
Any Erlang book recommendations?

~~~
jb3689
If you like the O'Reilly format then I liked Designing for Scalability with
Erlang/OTP. I thought it was light on the details beyond scaling beyond single
node OTP apps but it covers a lot of the built-in Erlang functionality pretty
well

[http://shop.oreilly.com/product/0636920024149.do](http://shop.oreilly.com/product/0636920024149.do)

