

Erlang/OTP 17.0 has been released - rachbelaid
http://erlang.org/pipermail/erlang-questions/2014-April/078563.html

======
lkrubner
Erlang is more than a language, it is a philosophy, and the language is merely
an expression of that philosophy. I strongly recommend that everyone read
"Making reliable distributed systems in the presence of software errors". This
is where Joe Armstrong describes the ideas that fed into the creation of
Erlang. I learned a great deal from it:

[http://www.erlang.org/download/armstrong_thesis_2003.pdf](http://www.erlang.org/download/armstrong_thesis_2003.pdf)

A few quotes:

\-------------------

The central problem addressed by this thesis is the problem of constructing
reliable systems from programs which may themselves contain errors.
Constructing such systems imposes a number of requirements on any programming
language that is to be used for the construction. I discuss these language
requirements, and show how they are satisfied by Erlang…

History:

1998 — Ericsson delivered the first AXD301. The AXD301 is the subject of one
of our case studies in Chapter 8. At the time of writing (2003) the AXD301 has
over 1.7 million lines of Erlang code which probably makes it the largest
system ever to be written in a functional style of programming.

…Telephone exchanges are expected to be extremely reliable: Typically having
less than two hours of down-time in 40 years

…We organise the software into a hierarchy of tasks that the system has to
perform. Each task corresponds to the achievement of a number of goals. The
software for a given task has to try and achieve the goals associated with the
task.

Tasks are ordered by complexity. The top level task is the most complex, when
all the goals in the top level task can be achieved then the system should
function perfectly. Lower level tasks should still allow the system to
function in an acceptable manner, though it may offer a reduced level of
service.

Programming a hierarchy of tasks needs a strong encapsulation method. We need
strong encapsulation for error isolation. We want to stop programming errors
in one part of the system adversely affecting software in other parts of the
system.

The essential problem that must be solved in making a fault-tolerant software
system is therefore that of fault-isolation. Different programmers will write
different modules, some modules will be correct, others will have errors. We
do not want the errors in one module to adversely affect the behaviour of a
module which does not have any errors.

…Our applications are structured using large numbers of communicating parallel
processes. We take this approach because [it] provides an architectural
infrastructure — we can organize our system as a set of communicating
processes. By enumerating all the processes in our system, and defining the
message passing channels between the processes we can conveniently partition
the system into a number of well-defined sub-components which can be
independently implemented, and tested.

Concurrent processes with no data sharing provide a strong measure of fault
isolation. A software error in a concurrent process should not influence
processing in the other processes in the system.

…In our system concurrency plays a central role, so much so that I have coined
the term Concurrency Oriented Programming to distinguish this style of
programming from other programming styles.

In Concurrency Oriented Programming the concurrent structure of the program
should follow the concurrent structure of the application. It is particularly
suited to programming applications which model or interact with the real
world.

Concurrency Oriented Programming also provides the two major advantages
commonly associated with object-oriented programming. These are polymorphism
and the use of defined protocols having the same message passing interface
between instances of different process types.

…In the real world sequential activities are a rarity. As we walk down the
street we would be very surprised to find only one thing happening, we expect
to encounter many simultaneous events. If we did not have the ability to
analyze and predict the outcome of many simultaneous events we would live in
great danger, and tasks like driving a car would be impossible. The fact that
we can do things which require processing massive amounts of parallel
information suggests that we are equipped with perceptual mechanisms which
allow us to intuitively understand concurrency without consciously thinking
about it.

When it comes to computer programming things suddenly become inverted.
Programming a sequential chain of activities is viewed the norm , and in some
sense is thought of as being easy, whereas programming collections of
concurrent activities is avoided as much as possible, and is generally
perceived as being difficult.

I believe that this is due to the poor support which is provided for
concurrency in virtually all conventional programming languages. The vast
majority of programming languages are essentially sequential; any concurrency
in the language is provided by the underlying operating system, and not by the
programming language.

The inability to isolate software components from each other is the main
reason why many popular programming languages cannot be used for making robust
system software.

…”It is essential for security to be able to isolate mistrusting programs from
one another, and to protect the host platform from such programs. Isolation is
difficult in object-oriented systems because objects can easily become
aliased. (An aliased object is one where at least two other objects hold a
reference to it.)” — Bryce

Bryce goes on to say that object aliasing is difficult if not impossible to
detect in practice, and recommends the use of protection domains (akin to OS
processes) to solve this problem.

…In a paper on Java Czajkowski, and Dayn`es, from Sun Microsystems, write:

“The only safe way to execute multiple applications, written in the Java
programming language, on the same computer is to use a separate JVM for each
of them, and to execute each JVM in a separate OS process. This introduces
various inefficiencies in resource utilization, which downgrades performance,
scalability, and application startup time. The benefits the language can offer
are thus reduced mainly to portability and improved programmer productivity.
Granted these are important, but the full potential of language-provided
safety is not realized. Instead there exists a curious distinction between
“language safety,” and “real safety”. ”

In this paper they introduce the MVM (an extension to the JVM) where their
goal is:

… to turn the JVM into an execution environment akin to an OS. In particular,
the abstraction of a process, offered by modern OSes, is the role model in
terms of features; isolation from other computations, resources accountability
and control, and ease of termination and resource reclamation.

To achieve this they conclude that:

… tasks cannot directly share objects, and that the only way for tasks to
communicate is to use standard, copying communication mechanisms, …

…Many of the features of the Tandem computer bear a striking similarity to the
design principles in the OTP system, and to the fundamental principles of
Concurrency Oriented Programming which where discussed earlier.

Here are two quotes from the paper, firstly the design principles on page 15

There is considerable controversy about how to modularize software. Starting
with Burroughs’ Espol and continuing through languages like Mesa and Ada,
compiler writers have assumed perfect hardware and contended that they can
provide good isolation through static compile-time type checking. In contrast,
operating systems designers have advocated run-time checking combined with the
process as the unit of protection and failure.

Although compiler checking and exception handling provided by programming
languages are real assets, history seems to have favored the run-time checks
plus the process approach to fault-containment. It has the virtue of
simplicity—if a process or its processor misbehaves, stop it. The process
provides a clean unit of modularity, service, fault containment and failure.

More modern work with object-oriented systems has also recognised the
importance of isolating software components from each other. In [21] Bryce and
Razafimahefa argue that is is essential to isolate programs from one another,
and from the programs which run in the host operating system. This, they
consider, is the essential characteristic that any object system must have. As
they point out in their paper, this is a difficult problem in an object-
oriented context.

~~~
_nato_
Yes, yes, yes. It really is a philosophy. Great summary on his incredible (and
quite readable) thesis.

------
mononcqc
Learn You Some Erlang has a new chapter on maps for the occasion:
[http://learnyousomeerlang.com/maps](http://learnyousomeerlang.com/maps)

~~~
rdtsc
Thank you! That book has been an amazing resource. One of the few ones I still
got in a dead tree format.

~~~
davidw
Even if you're not super interested in Erlang, I can't recommend this book
enough: it's clearly a labor of love, and very nicely written. It's worth
supporting that kind of effort.

~~~
rdtsc
Oh yeah. Erlang or not it teaches a lot of very useful principles -- fault
isolation, building distributed system, functional programming. Those can be
applied to general system design.

------
talklittle
Reference manual for Maps:
[http://www.erlang.org/doc/reference_manual/maps.html](http://www.erlang.org/doc/reference_manual/maps.html)

Manpage for Maps:
[http://www.erlang.org/doc/man/maps.html](http://www.erlang.org/doc/man/maps.html)

Older documentation for Maps (EEP):
[http://www.erlang.org/eeps/eep-0043.html](http://www.erlang.org/eeps/eep-0043.html)

Important note found on the reference manual page:

 _Maps are considered experimental during OTP 17 and may be subject to
change._

 _The documentation below describes it being possible to use arbitrary
expressions or variables as keys, this is NOT implemented in the current
version of Erlang /OTP._

 _Exceptions returns badarg instead of badmap, this will change in the future
releases._

~~~
bosky101
I especially like this property of maps that cannot be done in either records
or JSON ( had it been supported as a primitive)

    
    
        M4 = #{{"w", 1} => f()}.  % compound key associated with an evaluated expression
    

And of course self-referencing is a welcome addition (especially on the
terminal)

    
    
        Fac = fun(0) -> 1; (N) -> N*Fac(N-1) end
    

~B

~~~
mononcqc
The syntax would actually be:

    
    
        fun Fac(0) -> 1; Fac(N) -> N*Fac(N-1) end
    

Note that the name `Fac` is not visible outside of the fun's scope, and to use
the same variable name outside, you still need to bind it:

    
    
        F = fun Fac(0) -> 1; Fac(N) -> N*Fac(N-1) end

~~~
eurleif
Looks like this is pretty much identical to JavaScript?

~~~
mononcqc
Pretty much. Javascript has the same scoping rules for functions used as
expressions:

    
    
        var f = function fac(n) { if (n==0) { return 1; } else {return fac(n-1)*n; } };
        console.log(f(3));
        console.log(fac(3));
    

This will log '6' and error out on the last one (fac is undefined). This is
the same scoping Erlang uses for anonymous function names as far as I can
tell, with one difference: Erlang's function name is a variable that can be
used as such internally. In Javascript, the function name doesn't refer to a
variable per se:

    
    
        var f = function fac() { console.log(fac) };
        console.log(f);
        f();
    

Which will log 'fac()' and 'undefined'. In Erlang:

    
    
        1> F = fun Fac() -> Fac end.
        #Fun<erl_eval.44.71889879>
        2> F().
        #Fun<erl_eval.44.71889879>
     

Both instances (F and Fac within the right scope) return the same anonymous
function.

~~~
eurleif
Are you sure you didn't make a mistake, and confuse the undefined that f()
returned with an undefined output from the console.log(fac)? It looks to me
like the function is a perfectly normal variable.

    
    
        var f = function foo() { console.log('Me:', foo); }
        f();
        Me: function foo() { console.log('Me:', foo); }
        undefined

~~~
mononcqc
You're right, I've been fooled by Firebug merging in multiple identical output
lines together and missed the (2) it had put on the right hand side of the
screen. That makes a lot more sense.

------
mediocregopher
The addition of maps is really awesome. In my applications I've had records,
which are kind of like C structs, being passed around to represent state. But
if I need to add features the records become very constraining, because adding
another key to them invalidates any previously existing records when you do a
hot reload. So you have to jump through a bunch of hoops to get the records
migrated, or just restart the whole system, which is less than ideal.

With maps this whole class of problems goes away.

~~~
njharman
>With maps this whole class of problems goes away.

A friendly warning from long time dictionary (ab)user. They introduce their
own class of problems. Basically the strong vs weak type ones.

~~~
mediocregopher
Agreed. Erlang's maps have two different operators when setting values on
them, one if you care that the key already existed and one if you don't, so in
that way you can still somewhat enforce strictness of keys. But you can relax
that strictness when it comes time to roll out changes.

------
derefr
I see that, when you create several maps that have the same keys but with
different values, the key-set is shared between instances, and this is
basically what brings this particular implementation of maps up-to-par with
replacing Erlang records in their use-case.

I'm curious whether this efficiency-in-storage translates to effiency-in-
messaging, though:

• When you pass a map N times to another process, is the key-set copied N
times? (Not that bad if true, so I'm guessing so.)

• When you pass a map N times over the distribution protocol, is the key-set
_serialized and transmitted_ N times? (This'd be pretty bad if true.)

~~~
jlouis
The first one: Yes, it _has_ to. Otherwise the keyset would be shared among
processes and that is generally not possible in Erlang (Immutable binary data
being the exception)

The second one: yes and no. The format is described at
[http://erlang.org/doc/apps/erts/erl_ext_dist.html](http://erlang.org/doc/apps/erts/erl_ext_dist.html)
and maps are laid out as a serialized construction like you say. Two points
balances this out: There is an atom-cache which allow you to cache atoms and
make their representation small. And you can zlib-compress the data.

It is not set in stone yet and might even change in a later versions. The term
format is versioned so you can upgrade it later if need be.

~~~
derefr
Right, what I was asking in the second question is basically whether there
exists, or is planned, a keyset cache to go along with the atom cache. I'd
think that, if the atom cache is a good idea, a keyset cache would be good for
exactly the same reasons.

~~~
jlouis
If you do that, it is better to create an arbitrary caching construction of
subtrees and then reap the benefit by tighter packing of data in general. I
agree a keyset cache could be really nice to have going forward. It would
resemble what happens in-heap.

But the rule of the Ericsson OTP team is to get it correct before making it
fast.

~~~
derefr
Atoms and keysets both have pretty much the same caching semantics, though:
they get repeated over the wire with pretty good locality, and don't have
conflicting terms busting the cache in-between. That's not really true for
anything else, which makes me guess that an arbitrary term-branch cache would
be pretty useless.

~~~
jlouis
The idea of having arbitrary subtrees is to support an efficient compression
scheme. But you are indeed right that a keyset cache would be extremely
effective at limiting the size of maps.

------
wardb
Just updated the HOWTO for installation on OSX Mavericks with a working
Observer: [http://blog.equanimity.nl/blog/2014/04/09/erlang-r17-on-
osx-...](http://blog.equanimity.nl/blog/2014/04/09/erlang-r17-on-osx-with-wx-
and-a-working-observer/)

------
davidw
Here's the README:

[http://www.erlang.org/download/otp_src_17.0.readme](http://www.erlang.org/download/otp_src_17.0.readme)

------
dpezely
It's also worth calling attention to Lisp flavored Erlang (LFE), which has
come a long way recently. They've specifically addressed language artifacts
that would be more welcoming to someone coming from a strong Common Lisp
background.

For seeing where it's heading, read Duncan's website:
[http://technicae.cogitat.io/search/label/lfe](http://technicae.cogitat.io/search/label/lfe)
He gave an excellent presentation on this at Erlang Factory conference in San
Francisco last month.

Otherwise, the main website is: [http://lfe.github.io](http://lfe.github.io)

------
wcummings
Unfortunately, I won't be able to use a lot of the improvements in my
projects, since it'll take some time for everyone to adopt it. But maps and
dirty schedulers (!!!) are both very welcome.

~~~
ams6110
Do heed the warning in the README wrt dirty schedulers...

    
    
      Note that the functionality is experimental, and not
      supported. This functionality will be subject to backward
      incompatible changes. You should not enable the dirty
      scheduler functionality on production systems. It is only
      provided for testing.

~~~
wcummings
Yeah, I was speaking with one of the developers on IRC. The API can still be
changed, so now is the time to try it out and get my 2 cents in ;)

------
varg
Great ... maps is a worthy addition to the language :)

~~~
gargantuan
Get it at Erlang Solutions -- there are packages for Ubuntu, CentOS 6 and
others.

~~~
olgeni
After R16, we actually ended up packaging our own Erlang debs at $WORKPLACE
(libssl depencency mismatch in latest wheezy; mismatched sources for xmerl &
others; packages generated from a commit _different_ than the release, with
most app version numbers wrong; missing crypto applications because of some
unknown reason, etc...)

------
Ambrosia
roll on elixir 0.13

------
jeffdavis
So I guess it just missed the boat on ubuntu 14.04 LTS, right?

~~~
dpezely
There are instructions for using 'apt-get' and friends for installing current
releases:

[https://www.erlang-solutions.com/downloads/download-
erlang-o...](https://www.erlang-solutions.com/downloads/download-erlang-otp)

While R17 wasn't ready for this approach immediately upon the announcement,
I'd expect it any day since other pre-compiled packages have been made
available already.

Essentially:

wget [http://packages.erlang-solutions.com/erlang-
solutions_1.0_al...](http://packages.erlang-solutions.com/erlang-
solutions_1.0_all.deb)

sudo dpkg -i erlang-solutions_1.0_all.deb

sudo apt-get update

sudo apt-get install erlang

