Hacker News new | past | comments | ask | show | jobs | submit login

"First, it's important to take into account that ZeroMQ was intended to be a piece of infrastructure with continuous uptime. If should never fail and never exhibit undefined behavior."

There is one language/platform[1] I'm aware of designed for this type of system, and which achieves it beyond all others: erlang.

If Uptime is important, how do you handle it when you want to update your software under C++ or C? You have to bring the service down. Erlang supports updating software while its still running.

If exception handling is important (to prevent crashes) how do you handle it when there is a bug? Erlang has supervisors that not only help you manage exceptions but keeps the service going when an unhanded exception causes things to crash.

More importantly, intrinsic to the nature of the way you write erlang code, you eliminate a great deal of the possible errors. In C code you have to check to make sure things are in the right state while running your code. In erlang, your code doesn't run (the function isn't called) if the state isn't right to handle it. It flips this problem eliminating many classes of errors. Further there is OTP support for many types of processing (like state machines) that makes writing these types of solutions much easier.

Erlang is great for making sure no undefined behavior happens. Erlang is also great (as I understand it, this I haven't gotten into) for creating unit tests that can fully exercise the code across all possibilities.

Where does erlang suck? I don't think it sucks anywhere because it has solutions for all of its own problems. For instance, single node performance of an erlang program (which runs in the BEAM VM) is going to be lower than single node performance of a C program... but in erlang you can increase capacity nearly linearly simply by adding more nodes... while this is nearly impossible with almost all C/C++ programs without spending a lot of time working on it.

But where erlang is weak is that single node performance which might be really important[2], and if it is, you can then write your performance critical bits in C.

Thus the things that do the real work can be fast and in C but the things that keep the system going and distributed (where performance of this kind is much less critical, but performance of the operational kind is paramount) can be written in erlang.

So, I think the second mistake he made, and is still making, is thinking that the project has to be written in a single language.

If you use the best language for the job, then sometimes that should mean using two languages right?

Erlang is not a hipster language. No hipsters use erlang. It is 25 years old and boring and %99.9999999 uptime. Its ugly. Its not hot and fresh and new. But it is the right language for the job, when you've got a system that need to run across cores or nodes, or have very high uptime.

And really, it doesn't take but a couple weeks of learning. Then the syntax will be gorgeous and you'll be a better programmer for it.

[1] I say "I'm aware of" for a reason. There may be others as well suited when this is the primary goal, possibly even better suited. I don't know all languages. But I do have a pretty good survey started in the days when new languages were as common as YC batches and much more experimental than they are now. My point isn't to bash other languages but to promote one that isn't understood correctly by many people.

[2] I think this is the case a lot less often than people think. People benchmark things on a single node even when they know they are going to build a cluster of machines to run them, and then pick them based on this. Redis is really fast. Is redis distributed? Riak has a fully distributed in memory database (which means 10 32GB nodes means you can store 320GB in RAM if you want, rather than have 10 nodes all storing the same data in a cache, and thus much less data cached for the same number of machine with something like Redis.




Erlang is not a language for writing libraries that can be integrated into other runtimes. It is opinionated about concurrency, memory management, types, message passing, etc. That isn't inherently a bad thing, but it's the wrong choice for a library that is trying to offer in-process APIs in lots of different languages.

ZeroMQ is software that is more at the level of the Erlang VM itself. It's not an application, it's infrastructure. You wouldn't say that the Erlang VM should be written in Erlang.


Exactly, this makes no sense at all. 0MQ is a low-level library that higher level languages link into (see all the [bindings][1]); it's pretty much a veneer over plain sockets that adds a bunch of features. Why would you, or better, how could you write something like this in a language that has to run in a VM?

[1]: http://www.zeromq.org/bindings:_start


It is possible for C programs to update themselves with zero downtime. See nginx[1], "Upgrading Executable on the Fly". The trick is that fork()'ed processes (can) inherit open sockets from their parent, even after exec()'ing a new binary. This seems a bit delicate and involves some communication during the hand-off from the old to the new, but it does work. It's also possible to kill the "new" if there were any problems and instruct the "old" to take over again.

This is not to take anything away from Erlang, which is a very interesting system.

[1] http://www.nginx.org/en/docs/control.html


> This seems a bit delicate and involves some communication during the hand-off from the old to the new, but it does work.

MUDs routinely did this back in the day. Here's a README excerpt from Erwin Andreasen's widely used patch:

"Basically, for each playing descriptor, it saves the descriptor number, the player's original name as well as the host name (so we don't have to find that again). Then, it closes fpReserve, and exec's the MUD again. As exec preserves open file descriptors, players do not lose link. The MUD is executed with some extra parameters, so the new copy knows that it should reload some player files. It does that, and puts in the players at their old places."

http://www.andreasen.org/ftp/copyover-7.tar.gz


Just logged in to say this is the most awesome thing I've read in weeks. Why did I not know this earlier?!


> Where does erlang suck? I don't think it sucks anywhere because it has solutions for all of its own problems.

This is a very novice thing to say, for any language. I've found it takes me about 3-6 months of regular use to start seeing the warts of a language (and a few more to learn how to work around them).

Erlang seems pretty boss (I haven't used it myself) but this post reads like "just buy a Mac and everything will be easier!!!" Unfortunately that doesn't take into account the user's experience and needs. Good luck switching to a tiling window manager on a Mac.


Some of my reflections on Erlang's warts:

http://journal.dedasys.com/2007/09/22/erlang

It's from nearly 5 years ago, so they may have fixed some of those things.


If Uptime is important, how do you handle it when you want to update your software under C++ or C? You have to bring the service down. Erlang supports updating software while its still running.

Nginx has a hot update feature and is entirely written in C.


For that matter, Erlang is written in C.


I'm glad it has that feature, but you have to write it yourself. I wasn't saying it was impossible to do in C. Erlang has it built in, along with frameworks in OTP to make it easy to do.


If you had 1) looked up what ZeroMQ does, and 2) ever actually implemented a system in Erlang, you wouldn't dream of making a recommendation like this. It is completely irrelevant to the author of the article.

Further, Erlang makes no attempt to avoid undefined behavior. It provides mechanisms to restart pieces in a usable state to bring the system back into line rather than try to avoid or fix incorrect states.

Please don't post opinions about languages you have never done real work in. Your weekend project doesn't count.


And I think Erlang is unique in the respect that fault tolerance and uptime was the driving feature behind its development. It basically started with that -- they way I heard Ericsson would have contract that specified that anything beyond 4 minutes downtime would be billed at $10k / minute or so.

Then hot code reloading, supervisor structure, isolation of faults, actor model all sort of fell out of that.

And as Joe Armstrong also said in respect to single node performance is that "there is no free lunch". In other words at some point you'll have to sacrifice some performance if you want fault isolation.


> Erlang is also great (as I understand it, this I haven't gotten into) for creating unit tests that can fully exercise the code across all possibilities.

Specifically, Erlang has a good (and professionally supported, IIRC) port of Haskell's QuickCheck for exactly this sort of testing.


That's QuviQ's quickcheck. There's also proper: https://github.com/manopapad/proper


Erlang is cool. RabbitMQ core is written in Erlang. But I don't think zeromq could be written in Erlang and still meet the needs for which it was invented.


Yeah I wish people would just use Erlang instead of reinventing this stuff over and over again in their favorite language/framework/vm etc. When I think of all the work going into Node instead of Erlang it makes me want to cry.


I think for some project uptime is not as badly needed or not taken seriously enough as it was for Ericsson at the time. So for example during prototyping some engineer saw how Erlang was 5 times as slow as C. They told the manager and the manager said "oh well, clearly we should use C then".

If uptime is not the driving feature of the project. It will be very hard to add it later as the language chosen and the libraries have to have also been built with uptime in mind.


Not that this detracts at all from your point, but Erlang is 25 years old, and has been publicly available for 14. So it is relatively new compared to most of the other ones in current use.


However it has 25 years of real world use and that's what is important. I think Erlang used it for a serious industrial grade product -- AXD 301 switch (and maybe others I don't know about).


Yeah, I meant to be exaggerating for effect but it didn't come across that way at all, so I updated the post.


you are missing the point of ZMQ, it's a library not a service, writing it in erlang would completely defeat its purpose as comparatively very few people would be able to use it


The article is about undefined behaviour. Erlang won't help much with that.


Depends how you write your code. Actually Erlang is pretty good at avoiding undefined states (which can be thought of as undefined behaviour) by putting everything in an OTP framework. There's no custom "I'll just handle this earlier here in a tricky way" - if you stick to OTP you'll have message handlers and state machines. That's really awesome if you're programming network daemons. You can pretty much follow the state machine diagram and make sure that unless some progression is possible, it will not occur in practice.

Testing based on OTP service trees also makes the verification fit the possible transitions in a much closer way than trying to cover all the branches in an OO app.

So yes, I'd say it's actually quite a good response to an "exception handling all over the place" problem. You simply don't do that. Instead your exceptions finish at the FSM level and progress you to one of the error states on your diagram. You know exactly what state you're in at that point so the behaviour is pretty well defined at almost every step.

However zeromq is a library that was supposed to be portable, rather than a daemon itself, so from that perspective, it's a very bad fit for ZMQ.


What worried me about Erlang was that when a process dies and gets restarted, the message queue for the old process gets thrown away. Neither the sender(s) nor the new process can tell how many messages were lost.


If you care about reliability of the delivery, you can do a number of things: do a single call, wait for response and retry if needed, or store the messages in mnesia and only notify there's something to pick up (pull scenario with ack), or do a number of other possible things.

Lost messages are a fact of life really - the same will happen with any other system - either you persist the message and ack the reception on every stage or you risk dropping the queue.


Yes, that situation is usually handled by using an ETS table to store messages (essentially, you create an internal message queue for the supervision tree).

On the other hand, there's no guarantee the message arrived in the first place (especially when using multiple nodes) unless you use ack patterns.


Well, yes and no. I know more about OCaml than Erlang, but pattern matching and guards let you be very sure that a function is going to handle every possible input, and static typing means all code paths can be verified at compile time - there's no way into or out of a function without the program being in a valid state.


What about, say, a function that puts ten elements into a list that needs to get sent a receiver that only expects nine ? Or a network call that returns an unparseable reply?


In the first case, I wouldn't use a list for that, but a record/struct/class with 9 fields. In the second, I would use an Either/Maybe type, or raise an exception (these work differently in OCaml than in C++)


Firstly the article is about the use of C++ vs C in ZeroMQ (crossroads). Then the article is about systems having a high uptime and how C++ is not ideal for that.

Here is the quote:

> First, it's important to take into account that ZeroMQ was intended to be a piece of infrastructure with continuous uptime

So it is about uptime, that's the supposed goal of someone using ZeroMQ. C++ has undefined behavior and this, the author claims, can lead to more crashes and reduce uptime.

Then nirvana suggested that if uptime is needed, there is a better language and platform designed for it.


Impractical fanboyism that doesn't understand the software in question.

Poor attempts at proselytization like this do more to harm Erlang than help.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: