Hacker News new | comments | show | ask | jobs | submit login
Suture – Supervisor Trees for Go (jerf.org)
149 points by leef on Sept 17, 2014 | hide | past | web | favorite | 38 comments

It would be interesting to see what it takes to get this functionality (at least locally) into Haskell given that there are asynchronous exceptions... even ones with arguably greater safety than their Erlang compatriots.

A similar effort is happening with the Cloud Haskell program, but my understanding is that they're pouring a lot of effort into transmitting arbitrary Haskell functions over the wire between computers. This is pretty unnecessary for supervision and "Let it Crash"-style error handling alone.

That is a core primitive in Cloud Haskell but not the only one - you also send and receive data as in Erlang. distributed-process-platform provides Erlang style supervisors, gen_server, etc. It's all there and very much captures the features and spirit of OTP in a powerful static type system.

I'll definitely give it another look. I may have just misunderstood where the effort was being spent.

You may be thinking of the statics stuff being done with GHC 7.10.

I think I just missoveremphasized the importance of a particular paper on it that I read.

Somewhat different, but there is an attempt to implement BEAM in Haskell: https://github.com/gleber/erlhask

This problem (sync/async) is one of the things that is being solved in this project.

Oh, that's a bit disappointing. I thought Cloud Haskell was focused on developing actor-pattern abstractions like lightweight, concurrent threads and fault tolerance. Serializing functions seems like a real red herring.

You can totally do that in Cloud Haskell - serializing functions can be useful too but it's not the draw for me.

Great article. I especially enjoyed the clear descriptive problem framing and discussion about Erlang processes versus goroutines.

Agreed - I think this was one of the best written descriptions of a problem and then a solution set. Enough so that even someone only quasi-technical could make sense of a good portion of it.

This was interesting simply for the explanation of Erlang systems, but as for the project itself I currently use Supervisor (http://supervisord.org) for this purpose, and I've been pretty happy with it.

Any reasons to make the switch?

For when you want to supervise goroutines instead of processes (Presumably because processes are consuming too many resources).

Exactly. One of the big ideas of Erlang is to have many lightweight processes (processes in the VM, not OS).

As an example say you have a chat system, you will have a process for each user [0]. If you have a million users, you will end up with a million processes. The supervisor tree watches these processes, so if one crashes a single user may get logged out but everybody else will be unaffected.

[0] In reality you'll probably have it even more fine grained than that, so users will end up with multiple processes.

Why not just use Erlang?

I'm comfortable with it, but none of my team wants to use it.

Even after 7 years of experience, it is very klunky to program in, and in reimplementing the Erlang program I had in hand, I immediately added some features that were very hard to implement in Erlang. And note the architecture of the code didn't actually change, this isn't one of those cases where the second version was also radically different than the first so no comparison is possible, it was actually hard to implement in Erlang. Erlang is a very klunky language to work in. Raw Java might be klunkier, but it's easily the klunkiest of the languages I use.

Erlang's clustering, which my code heavily depends on, just sort of breaks sometimes, on machines that are on the same switch, with every bit of hardware switched out over time. It is very opaque to figure out why it isn't working, or how to make it work again. Mnesia breaks even more often on loads that shouldn't be a problem. After 7 years of working with it, at this point even if the solution was just to turn on this one flag somewhere and all my problems would go away, I still would not consider that a positive.

I don't "hate" it, but I've been waiting for years for a more conventional language that had pervasive microthreading in it that I could use for work. Go is not quite it... I'd still like the goroutines to be completely isolated, and I've got a handful of other quibbles... but it's close enough that I might be able to get out of the hell of trying to multiprocess in most other languages, while being something I might actually be able to get adopted. In general, the company I work for has not been overwhelmed by the effectiveness of Erlang.

In practice, the biggest loss of fundamental capability is loss of true asynchronous exceptions, and, well... in practice that's only an annoyance, not an OH MY GOSH I WILL NEVER USE A LANGUAGE WITHOUT ASYNCHRONOUS EXCEPTIONS sort of thing. In most cases if you're throwing an async exception you've really already lost and now we're just discussing exactly how you're losing.

Every language student should study Erlang, and learn from it, and possibly use it. It's a solid language. And there's nothing wrong with trying to port some of that solidity elsewhere. (I'd pick a real DB over Mnesia, though.)

> In most cases if you're throwing an async exception you've really already lost and now we're just discussing exactly how you're losing.

That is a exceptionally quotable. :)

Have you taken a look at Akka (with Scala)? You sacrifice code hot-swapping and I think little else, but you get JVM performance and one of the most extensible non-Lisp and safe non-Haskell languages in existence.

Or Akka with Java

Yeah, but the reasons for choosing Java over Scala are mostly organizational. If your organization can tolerate Erlang it can tolerate Scala.

But his can't tolerate Erlang, that's why he used Go.

Sorry, kinda missed that.

It seems he wanted a static type system, for starters.

There's a running joke in the Erlang community that every Erlang programmer will at some point try and re-implement Supervisors and do it horribly wrong. This isn't because the person is a bad programmer, it's just easy to underestimate the amount of time and incredible programming that has gone into Erlang's supervisors.

By sticking with Go, the author gets to keep the static types but likely gets faux-Supervisors at best.

I describe exactly what I get. Whether they're "faux" depends on your definition.

No, seriously, that's really profoundly true. Erlang's definition isn't the only one. But you're free to take it. If you can bend a bit, though, these may not be "faux".

One thing that's easier when it comes to implementing supervisors in not-Erlang though is that the vast majority of languages have something better than "behaviors" to work with, which are bizarrely klunky things. I actually implemented something more like a "gen_server" initially, but it wasn't a win. Some of that "incredible programming" is essential complexity (in Fred Brooks' terminology), but some of it is accidental complexity. Plus Erlang had to build it from true, total scratch; Go is a pre-existing runtime, and "defer" is simple, yet powerful.

Did you try to implement your own behaviours? Or processes that aren't behaviours as such but still function correctly in supervision trees and in applications?

What I found is that if you implement Serve() and Stop(), honestly, you're done, in the world of Go. If you want a state machine, just write it. Many of the things I'm supervising are goroutines that wrap an API around a channel that implements a server.

While not exactly static typing, you can add type annotations in Erlang and use dialyzer as a step in compilation to catch typing bugs.

In the spirit of rubiquity's post, I'll take a faux supervisor tree and static typing over a "real" supervisor tree (sorry, gotta scare quote it) and faux static typing.

It should be pointed out specifically that Erlang's type system is incredibly, annoyingly weak, more so than merely "weak typing". It's a language that doesn't permit any sort of user-created type whatsoever. There are good reasons for this which would be its own interesting blog post, mostly related to the need for a clear serialization for cluster communication, but I think those good reasons have been superceded by later work in the general programming world on serialization of objects and the tradeoff is wrong in 2014. (But, no sarcasm, a good one when it was created. It is no real criticism of a language to observe that it couldn't use lessons learned only a decade after it was created.)

> It's a language that doesn't permit any sort of user-created object whatsoever.

I think you are misusing "object" to mean something like "class" or "type" (the two not really being the same thing except in some statically-typed, class-based, OOPLs); Erlang quite obviously allows user-created objects.

Yes, it should be "type". I've edited it, so let the record show that I did say that at first.

And of course you can use what exists to build "types", such as a dict. But it's still just a record of a certain format rather than a distinct "type"; there's no way in the core system to assert that you have such a "dict", beyond having a record.

Sure, but this isn't all that different than what you get with "tags" (dynamic types) in a dynamically-typed language, even if some of those languages have some construct for runtime type testing (Erlang doesn't, but has pattern matching which seems to provide more than dynamtic-type-testing-specific functionality would.)

If the complaint is just Erlang-lacks-static-typing, I can agree that that's a fact, but I don't see how what Erlang does give you is less useful (even though the particular structure is different) than what you get in other languages that lack static typing.

"but I don't see how what Erlang does give you is less useful (even though the particular structure is different) than what you get in other languages that lack static typing."

Well, this gets into quasi-religious debates about type systems. All I can really do is point you to the Python or Ruby class systems and ask you to compare it to Erlang. And also observe that I'm pretty tired of dynamic typing in general, and Erlang is even farther in the direction that I'm tired of than Python/Ruby/etc.

Erlang has static types, they're just (a) always open and (b) always optional. It's more or less the most dynamic static typing regime imaginable. In personal use, it's highly valuable from a documentation standpoint, but does almost nothing for safety or restricting abuse of power.

Here's a recent Erlang type annoyance:

Given two tuples with 3 integers, {X, Y, Z}, without knowing it a priori, you can't tell which one is an erlang:now() and which is an erlang:date().

It's exactly static typing, just a very loose system which allows a lot of holes.

I did something similar in Python, using Redis for the "operating system" and Linux processes. The system managed wide sets of parallel I/O operations with second-level latency. Worked great, albeit orders of magnitude slower than something like this. If you keep the supervisor on the same machine, you can do async exceptions via kill.

Any pointers on what blogging setup is being used at jerf.org for writing this post? I found the page pretty tasteful.

It has all the signatures of a static site generator.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact