
Erlang/OTP 20.0 - okket
http://erlang.org/doc/index.html
======
rdtsc
Some things I like from the new release:

* Dirty schedulers: This allows easy integration of blocking C-based libraries. So for example can wrap something like RocksDb and make it available to the rest of the VM easier. Or libcurl and others.

* DTLS : This lets it talk to WebRTC clients

* Erlang literals are no longer copied when sending messages : This is kinda of a sneaky one. By default (with exception of large binaries) Erlang VM usually copies data when it sends messages. However, in this case module literals (constants, strings, etc) will be another thing that's not copied. There is a hack to dynamically compile configuration values or other tables of constants as a module at runtime. So if you use that hack, you'd get a nice performance boost.

* code_change, terminate and handle_info callbacks optional in the OTP behaviors. This is very nice. I always wondered why I had to write all that boiler plate code.

Also here is a detailed list of changes:

[http://erlang.org/download/otp_src_20.0-rc2.readme](http://erlang.org/download/otp_src_20.0-rc2.readme)

~~~
mtanski
> * Dirty schedulers: This allows easy integration of blocking C-based
> libraries. So for example can wrap something like RocksDb and make it
> available to the rest of the VM easier. Or libcurl and others.

Silly question, but shouldn't elang have some good & highly concurrent http
libraries?

~~~
micmus
Dirty schedules are about functions implemented in C - called NIFs (Native
Implemented Functions) - a sort of FFI you'd find in most languages.

Because of preemptive nature of Erlang, doing lengthy work in those functions
can destabilise the system (the "magic value" is said to be around 1ms). That
is because C functions can't be preempted in the middle of execution like
Erlang ones can.

Using dirty schedulers lifts this time limitation, but gives a higher constant
overhead when calling a function on a dirty scheduler, since it means
switching OS thread. This tradeoff, however, is perfectly acceptable for a lot
of cases.

And yes, Erlang has good & highly concurrent HTTP libraries implemented in
Erlang.

~~~
eugene_pirogov
A question regarding dirty schedulers: does it still mean that when called C
code crashes during execution, entire Erlang VM will crash as well?

~~~
micmus
Yes. That's not changed. All NIFs (regular or dirty ones) are executed
directly in the context of the VM. A safe option would be a port - a regular
program where you communicate through stdin/stdout. Ports allow representing
such program (running in a separate OS process) as something equivalent to a
native Erlang process.

There's also work on supporting writing NIFs in Rust, which gives some degree
of additional safety. The relevant project would be:
[https://github.com/hansihe/rustler](https://github.com/hansihe/rustler)

------
marianoguerra
if you want to get started but don't know where this is a good place to start:

[http://spawnedshelter.com/](http://spawnedshelter.com/)

there's a online course by Simon Thompson from the University of Kent that
started 2 days ago, you may be able to join:

[https://www.futurelearn.com/courses/functional-
programming-e...](https://www.futurelearn.com/courses/functional-programming-
erlang)

if you like the ideas but want to try something different there are
alternative languages that run on the Erlang VM:

* Elixir with a ruby-like syntax: [https://elixir-lang.org/](https://elixir-lang.org/)

* LFE (Lisp Flavoured Erlang): [http://lfe.io/](http://lfe.io/)

* Efene with a python-like syntax: [http://efene.org/](http://efene.org/)

and one in development but already looking really interesting: Alpaca, an ML
inspired language: [https://github.com/alpaca-
lang/alpaca](https://github.com/alpaca-lang/alpaca)

~~~
dbcurtis
Thanks! I came here to ask the "How do I Erlang?" question. Erlang has been on
my radar for a long time but I've never taken the time to dig in. 85% of my
code output is obscure protocol stacks in Python+Twisted, the other 15% is
real-time embedded C over an RTOS. Even if I can't use Erlang in production,
I'm sure I'd find food for thought in the concepts.

~~~
digitalzombie
[http://learnyousomeerlang.com/](http://learnyousomeerlang.com/)

That book is a good place to start.

> Even if I can't use Erlang in production, I'm sure I'd find food for thought
> in the concepts.

Oh boy are you going to be in for a treat. The concept for no loop construct
other than recursion and recursive thinking was worth it for the little time I
dabble in Erlang.

~~~
equalunique
As an erlang newcomer, the non-loop seems to be one of the most intriguing
concepts, IMO.

~~~
marianoguerra
You still have list comprehensions, map, filter, fold and foreach using higher
order functions

------
sho
My favourite feature - shell history finally in core! That's been a possibly
irrational but personally annoying omission for years, and required messy
hacks to get around - not a good beginner experience for what I consider to be
pretty basic functionality.

It's enabled by an envar as described by the hero who cleaned up the hack and
ported it here: [https://github.com/ferd/erlang-
history](https://github.com/ferd/erlang-history)

~~~
mononcqc
Bit of history on the feature.

The hack was an old thing I had written at a hackathon over half a decade ago.
It had little cost to regulars who had installed it at least once before. It
relied on DETS and had transient issues with corruption of history files, but
nothing major enough for anyone to take notice or get mad about it.

I myself ended up working on other projects once that one was good enough
(including the search function with ctrl+r in the shell -- that wasn't there
at first, and then things like rebar3 and other side projects), and nobody
really ever took over or reimplemented a better shell history around it, nor
took the time to port it to OTP.

I had also asked for help in the past for a better implementation that didn't
rely on DETS, but not a lot of people were interested or knew how to proceed
any better than I did. Eventually I just got tired and reimplemented it with
the disk_log module and some log fragmentation and rotation to avoid dropping
the whole file every time a limit was reached. It took a few weeks of review
to get it in and so far nobody has complained about it.

I think the overall lesson is that the moment you have a quick hack to do the
job, regulars get used to them, but forget about the beginner's experience
where what you have to do is apply _all_ the hacks at once and dear lord does
it ever look janky then.

~~~
sho
Thanks very much for your work - really makes a big difference to me and i'm
sure many, many others. Beer's on me if you ever make it to SEA!

------
marianoguerra
Also the docs got a facelift: [http://erlang.org/doc/](http://erlang.org/doc/)

Now that we are talking Erlang:

What's missing in Erlang that would be valuable to you?

What are the biggest pain points right now on the Erlang ecosystem that makes
it harder for you to try it/adopt it?

~~~
digitalzombie
> What are the biggest pain points right now on the Erlang ecosystem that
> makes it harder for you to try it/adopt it?

Personally, Erlang solve stuff that is very niche and I don't have any project
in those stuff.

Elixir web frameworks really help in term of practicing the language.

Erlang had cowboy, etc.. but it wasn't close to the other MVC language. So
practicing Erlang to get real experiences in it was hard.

Maybe there should be a page of examples of other problem Erlang can solve so
I can decide which problem I like to start a project on? Cause so far when I
think of Erlang it's just concurrency, back end protocol, etc... But I'm sure
there may be other problems it can be address that I don't know about and that
I would like to start a trivial project on it.

~~~
klibertp
> Erlang solve stuff that is very niche

Well, it's true that Erlang supports some strange, esoteric (at least,
esoteric for people outside of telecom) protocols.

But it supports all the "normal" stuff also, as you write:

> Erlang had cowboy, etc.. but it wasn't close to the other MVC language.

Cowboy is actually pretty decent a framework for web development. It's not
Rails or Django equivalent, it's similar to something like stripped-down
CherryPy, which may be regarded as minimalistic, but it's perfectly
functional. And fast. And WebSocket handling is pure bliss.

So, Cowboy doesn't support MVC out of the box, but you can implement this
pattern on top of Cowboy in ~30 loc.

> So practicing Erlang to get real experiences in it was hard.

Erlang is different. From its Prolog roots to its concurrency primitives to
immutability and tail-call elimination - it's built to be different.

It's not exactly surprising that you can't use it as freely as you'd like
immediately after learning it. It's expected. The only thing I can tell you
now is that the problem disappears with time. You need to practice Erlang
fundamentals for a bit and slowly advance to more complex topics.

In my case, it took me a year to become proficient with Erlang: the main
language was a bit of a hassle, but its std library and OTP and ecosystem in
general took even more effort to grok. OTOH, after a year of learning and
practicing, I was able to write a proof-of-concept web apps in a day,
including learning Cowboy. The "proof-of-concept" was fast and solid enough
that it reached production stage and ran there for 3 years until finally
replaced by some other tool.

So I'd say it's worth the effort, but of course YMMV :)

------
di4na
Something that most people seems to have missed

> The non SMP Erlang VM is deprecated and not built by default

In 2017, most languages are still fighting to get a multicore implementation.
Erlang is ditching their single core one.

------
desireco42
This is probably better page to look, as I don't actively look at Erlang, but
as Elixir developer I am very interested in it.

[http://www.erlang.org/news/114](http://www.erlang.org/news/114)

~~~
cmpaul
Looking for work? :)

[https://jobs.lever.co/hellosign/39ea6c41-8ff8-4c93-b0fd-4a62...](https://jobs.lever.co/hellosign/39ea6c41-8ff8-4c93-b0fd-4a6224ef29ec)

~~~
desireco42
Yes :) I am in Chicago, I would be happy to do remote work.

~~~
Arcsech
I just want to echo this: I work in Elixir on side projects at least, and
HelloSign sounds like a cool place to work! But no way I'm moving to
California.

------
chx
Let's note that Elixir is already compatible with it. Phoenix/Elixir, I feel,
is the future of web apps.

~~~
pmontra
I used to think the same but I'm not sure about it now. We have plenty of
managed backends now and they usually support Node, Python and a few other
languages [1]. BEAM ones are never among them. Furthermore the deployment
story now is all about docker, which is at odds with the mosts advanced
features of BEAM (hot reload, but I guess few people use it.)

So, it might be the future for fully self managed web apps, but developers are
moving in another direction now. That damages the mindshare.

[1] Basically the owners of those backends have a voice in which languages
succeed and which not. They'll never support more than a handful of languages,
because of the maintenance costs.

------
thedaemon
Just an FYI Wings3D [http://www.wings3d.com/](http://www.wings3d.com/) is
built with Erlang. It is a Symbolics style 3d Modelling application using the
Winged Edge data structure.

------
brian_herman
Is there like a why's poignant guide to ruby for
erlang?[http://poignant.guide/book/](http://poignant.guide/book/)

~~~
shadeless
There's "Learn You Some Erlang for great good!"
([http://learnyousomeerlang.com/](http://learnyousomeerlang.com/)), which is a
great (the best?) resource for learning Erlang.

~~~
hordeallergy
I found it too densely informative. After the 2nd or 3rd attempt I ditched it
and just jumped into programming. Personal preference possibly. I think
erlang's difficulty is exaggerated. Just jump in - the immediate little wins
will lubricate the process.

~~~
derefr
"Densely informative" is the right way for a _textbook_ to be, I think. You're
not meant to read LYSE or books like it straight through; rather, each
individual paragraph is dense enough with facts that after every one, you
should stop and tinker with a toy example or two at the REPL to collect your
thoughts and make sure you understand what's going on.

If you imagine someone teaching Erlang with LYSE serving as the actual
textbook, they'd do what is done with most textbooks: split each chapter into
rather small segments, and have you do exercises after each one to
"interactively" absorb the knowledge.

------
lokedhs
I have tried on multiple occasions to learn and use Erlang for some different
projects.

My process has always been stopped by a few specific things, and I end up
going back to Common Lisp, which tends to be much easier to get to work.

My first problem is to make sense out of how to actually run a server
application. Manually compiling source files and run functions is one thing,
but trying to actually set up an application with all the monitors and stuff
is really annoyingly hard. Even following step-by-step instructions resulted
in errors that I didn't understand as soon as I tried to do anything outside
what the tutorial showed.

The second issue that I have never managed to properly solve is how to use
libraries. In particular, I have wanted to use the client libraries for
RabbitMQ and CouchDB (both written in Erlang, so you'd expect it to be
simple). The instructions how to install the libraries usually doesn't match
the thing you actually download.

I've also tried to use the EDTS in Emacs, which is pretty nice, but as soon as
I try to use a library, it can never find the hrl files.

I really want to use Erlang, but the difficulty of actually getting started
with a real project as opposed to simple tutorial stuff has been extremely
frustrating.

~~~
JamesUtah07
Try out elixir. It runs on the Erlang VM but is easy to get up and running and
has a great community. It's much easier to add external libraries and uses its
own package manager.

~~~
lokedhs
Thanks for the suggestions (you, and the other two people mentioning the same
thing). I have just started looking at Elixir, and I'm hoping that will work
better.

Am I just to assume that Erlang deployment really is as complicated and poorly
documented as I thought, since no one suggested any documentation, but three
people suggested I switch to Elixir.

~~~
klibertp
In my experience, Erlang doesn't have a standard way of doing a deployment.
There are 3rd party apps like Rebar and others, but they are both not
available by default (after a fresh install) _and_ seriously underdocumented.
The result of this, IMHO, is that everyone builds their own build and
deployment system, using shell scripting, Make or Erlang itself.

Elixir includes Mix - a standard task runner - and Hex as part of the
distribution. It also provides a central pkg repository
([https://hex.pm/packages](https://hex.pm/packages)). It's much, much easier
to start a project and pull all your dependencies.

I guess you could use Hex and Mix with pure Erlang projects, but I haven't
tried this.

~~~
marianoguerra
rebar3 will become the standard build tool for erlang, it's already in the
erlang organization
([https://github.com/erlang/rebar3](https://github.com/erlang/rebar3)), most
people nowadays use either rebar3 or erlang.mk
([https://erlang.mk/](https://erlang.mk/)), both work really well, at least
for me.

------
atemerev
Erlang is awesome. The only problem preventing me to use it where I want to is
terrible file I/O performance, especially on writes. I tried to Google the
solution, but it seems that there isn't anything generally accepted for the
moment.

~~~
marianoguerra
something specific? did you tried raw file handles?
[http://erlang.org/doc/man/file.html#open-2](http://erlang.org/doc/man/file.html#open-2)

~~~
atemerev
Yes, and delayed_write as well.

I tried to port a (very simplistic, but fast) market data append-only database
from Scala to Erlang. In Scala, I have no performance issues, but the code is
unnecesarily complex to my taste.

In Scala, I am getting around 100000 events per second, with great additional
optimization margins (memory mapped files are great). In Erlang, it barely
works with few hundred events per second.

Right now I have googled that disk_log in Erlang is fast enough, but it uses
Erlang's own internal binary format (there is an option to plug custom codecs,
but it is wonderfully under-documented, to say the least).

This looks strange to me, because Erlang is well-optimized regarding the
network IO. What's the difference for file IO?

~~~
derefr
> Erlang is well-optimized regarding the network IO. What's the difference for
> file IO?

IIRC, async network IO is handled by the VM's scheduler threads each just
calling a non-blocking select() on the fdset of sockets it holds port()s for
every once in a while. This is because TCP/IP implementations are pretty much
guaranteed to expose a non-blocking select() for sockets, and do in every OS
Erlang is written for.

Disk IO, on the other hand, is done by throwing the calls over to special
async IO threads, which have to send responses back to the scheduler that
wants them using (I think zero-copy) IPC. This is done because _not_ all OSes
have the equivalent of non-blocking select() for file handles (i.e. what you
get from using fcntl(2) + read(2) in Linux.) So Erlang's disk IO BIFs
fundamentally require context-switches and/or NUMA messaging.

You get to avoid this if you use your own NIFs—which have the lovely property
of running in the calling scheduler by default, with the ability to decide on
each call whether the workload for this particular call is small enough to
perform synchronously, or whether it should be scheduled over to a dirty
scheduler, blocking the Erlang process and yielding the scheduler. In other
words, this is like the disk IO BIFs in the worst case, and can be a lot
faster in the best case. It's a lot like NT's "overlapped IO" primitives,
actually.

You could also use higher-level BIFs. There is a reason Mnesia is implemented
in terms of special BIFs (DETS) rather than DETS being a functional layer
implemented using the disk IO BIFs. Mind you, DETS probably doesn't have the
API you're looking for, but there are a number of NIF libraries that plug low-
level "storage engines" into Erlang as NIFs that provide equivalent
convenience:

• [https://github.com/cloudant/nifile](https://github.com/cloudant/nifile)

• [https://github.com/basho/eleveldb](https://github.com/basho/eleveldb)

• [https://github.com/gburd/lmdb](https://github.com/gburd/lmdb)

And there's also always the option to do what Erlang-on-Xen does: forego
anything that "devolves into" disk IO entirely, implementing the file module
in terms of network IO—the p9 protocol in their case. This is actually likely
the lowest-overhead move you can make if you're going to be running your
Erlang node on a VM that would just be talking to a virtual disk mounted from
a SAN anyway. Instead of the OS mounting the disk from the SAN and Erlang
talking to the OS, you can just have Erlang talk to the SAN directly using
e.g. iSCSI.

~~~
twic
If you are writing sequentially to a local disk, perhaps it would make sense
to create a tiny shim process that listens on a socket and writes to a file.
You could do that in C, or perhaps even as a socat invocation.

~~~
derefr
Do note that this _should_ have about equivalent performance to Erlang's disk
IO BIFs, as this is essentially what they're already doing—the async IO
threads are the "tiny shim process." (The difference mostly comes down to the
VM having an efficient internal IPC protocol to them, and the runtime being
able to control their CPU core affinity.)

Note also that if you turn ERTS's async IO threads _off_ (set the number of
them to 0), you should get improved throughput on disk IO tasks, as you're
then forcing the scheduler threads to do the disk IO themselves. Of course,
this trades off against latency, because the fallback here is blocking calls.

Sadly, the async-thread-pool architecture has meant Erlang has had no reason
to implement anything like Linux's kernel AIO support. (I wonder if they'd
accept a patch that specialized the efile driver for Linux, the way it uses
overlapped IO on win32...)

------
wtian2000
Somewhat irrelevant, but I noticed that the copyright line at the bottom still
says "Copyright © 1999-2016 Ericsson AB". Should it be updated to 2017?

------
lngnmn
It seems that the source build is broken. At least a straightforward attempt
to build otp from github sources with latest clang/lld has been failed.

------
2017throw
My favourite DSL!

