Hacker News new | past | comments | ask | show | jobs | submit login
Stuff Goes Bad: Erlang in Anger (2016) (erlang-in-anger.com)
293 points by mutor on Sept 6, 2017 | hide | past | web | favorite | 25 comments

Still a recommended book if you need to handle Erlang systems. Either as a developer or as an operator.

* Still good if you work with Elixir

* Good if you have, say, RabbitMQ or Riak in your stack

* Fred knows his stuff.

Most great Erlang resources, like this and Designing for Scalability, are very much recommended for Elixir programmers. It'd be a shame for them to disregard some of the best OTP resources just because they're in Erlang.

Just a quick +1 for "Designing for Scalability", as an avid Elixir programmer. It's a great book on the Erlang/OTP fundamentals which underlie Elixir and Phoenix.

I'm going to go against the stream an say -1 on "Designing for Scalability". Not that it isn't a good book, but the title doesn't really reflect the contents. It's more of a beginners book on Erlang/OTP than of scalability.

Do you have a recommendation for a more advanced book on that particular topic?

I'd also love to see something from the other angle: taking a program that's already an instantiation of a complex, distributed architecture, written in e.g. C, and then rewriting it in idiomatic Erlang (or Elixir—especially with GenStage/Flow stuff) to reduce its code size while retaining the same distributed-system properties as its ancestor system.

Even if you don't need to handle Erlang systems, there is still enough to learn if you need to build reliable systems.


Remsh and recon are tools I use almost every day.

Chapter 3 planning for overload is useful in general not just for Erlang systems, and is something that is not given enough consideration usually.

I've since written a fairly complete blog post only on overload management of asynchronous systems at https://ferd.ca/handling-overload.html -- it hasn't made it in the book, though.

Wow, wonderful post that I hadn't read yet.

In particular the 'fuse' type stuff is really important when you have a real world system where "let it crash!" is not really ok, because you have, say, a user interface, or a web server that should not be part of everything just crashing. In particular, something I've seen: if your Erlang application talks to a database, and the database goes down for some reason, it may be desirable to have the whole application not go down with it!

Thank you!

That is an awesome site. Does anyone know if there's a similar resource for JavaScript that gives high and low level information about the language?

There's a "thanks" section at the bottom that linked to http://superherojs.com/ which seems to be the same thing for JS : )

you don't know javascript is fantastic at the more advanced stuff.


Previous discussions:

https://news.ycombinator.com/item?id=8330501 (3 years ago, 54 comments)

https://news.ycombinator.com/item?id=8979267 (3 years ago, 21 comments, dupe)

Erlang (even though I only use it via RabbitMQ) forever changed how I think of reliable systems.

Hardware has always been a story of reliably operating unreliable parts. It was eye-opening to see the same concepts applied to software.

Fail fast, fail early!

(Also, Erlang did a really good job of lightweight threading, and how that interlaces with things like a cluster of servers.)

> Hardware has always been a story of reliably operating unreliable parts. It was eye-opening to see the same concepts applied to software.

Walter Bright has been advocating this for some time:


I've heard it mostly in military contexts, like "That ship has never fired her guns in anger", but otherwise not.

I don't use Erlang or Elixir but I love the idea of this book. Are there any similar troubleshooting guides for other languages?

There is a sense in which it is not possible to write a similar book for other languages. Erlang's "Let it crash" design philosophy is more or less unique among languages used as much as more than Erlang. This means the definition of 'trouble' which requires troubleshooting will tend to be different when trouble is not an error in business logic. Likewise the details of implementing solutions will be different, for example handling a high frequency of crashes by making 10x the number of concurrent actors available to the system.

{A Love Song to Erlang} I think Erlang is a beautiful piece of engineering design as engineering design. I would not say that about any other language and find other languages beautifully designed for other reasons. But Erlang is an engineer's tool designed based on engineering principles. The abstractions are engineering abstractions...but with garbage collection.

There is a book like this for Ruby (on Rails?) which I believe is the inspiration for this Erlang book. I just can't seem to find it at the moment.

It was about improving old/legacy/poorly-built code bases that are in a bad shape and need to be 'rescued'.

I have not read any Rails book ever and they could therefore not be the inspiration for this one :)

The true rationale behind the book was to give a better tool to train new coworkers that could help us operate the Erlang routing and logging stacks back when I was still at Heroku, and at the same time to provide similarly useful resources to other people maintaining similar software in the broader community.

Most of the operational aspects of Erlang were a bit of black magic, knowing where to dig through experience, having read the right kind of docs, and everyone would carry little gists of incantations and anonymous functions to run on production servers to help figure out what may be going oddly on there by correctly prodding at the VM.

I wrote the 'recon' library along with Erlang in Anger so that instead of just being black magic & oral tradition, people would have a better set of well-encoded practices to help approach the debugging and optimization of running systems.

Fair enough, the landing page and description just seemed similar to the one I remember for Ruby, that's all. Although that was more about rescuing a project rather than overcoming common barriers in a programming language.

Sounds like a great basis for an advanced programming book either way.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact