
Nanomsg – MIT-licensed ZeroMQ alternative - rumcajz
http://www.freelists.org/post/nanomsg/nanomsg-01alpha-released
======
aktau
I personally like the fact that the codebase (up until a few months ago at
least, when I last looked) is very clean and very understandable.

This probably has to do with both the transition from C++ to C, and the fact
that Martin (the author) has a better idea of what he wants to make. In other
words, he is (painfully) aware of the errors he made in his ZeroMQ and is
doing it right from the first time in nanomsg. This leads to clearer and more
consise code. The reduction of strange bugs and perhaps better performance
adds a lot of icing on the cake as well. I look forward to using nanomsg in my
future projects!

EDIT: github link,
[https://github.com/250bpm/nanomsg](https://github.com/250bpm/nanomsg)

~~~
malkia
I've been totally conversed back to "C" from "C++" after working on a port of
Metal Gear Solid to PC in 2000. I've had no problems understanding and reading
through the code the Konami developers have written, it was using some kind of
generic OO system in "C" with TCL-like passing of arguments (object* obj, int
argc, const char*argv[]) - and passing text messages. Yes that was okay even
on Playstation1 I guess.

Now I still prefer C++ at work for certain things as I'm simply too lazy to do
it correctly in "C" \- give me STL, Qt, any UI framework written in there, and
it's easier to use. But for core/system libraries (zlib, png, jpeg, tiff, zmq,
libpq, etc.) I'd rather have it in "C", at least on the interface part, but
I've seen what hiding of C++ or .NET can do with a crazy debugger :)

------
masklinn
It's important that the author of nanomsg is also the author of ZMQ. One can
see nanomsg as something built on the lessons he learned from ZMQ, assuming he
can avoid second-system syndrome.

~~~
rumcajz
I've worked on OpenAMQ before ZeroMQ. And lots of proprietary stuff before
that. So it would be more like dozenth-system syndrome :)

~~~
alexchamberlain
ZeroMQ isn't really a message queue though...

~~~
rumcajz
Traditional message queues are actually composed of two rather different
subsystem: the queue itself (basically a database) and the networking.

I would say we are on the path of separating the two. ZeroMQ does away with
the centralised server holding all the queues. It still has local queues
though.

nanomsg goes even further and gets rid of queues entirely. What's left it
networking buffers, typically implemented at L4 of the network stack.

~~~
derefr
So, then, what do you get by using nanomsg? Is the value proposition "BSD
sockets but less lousy"?

~~~
rumcajz
Nope. It's a platform for delivering packaged and reusable distributed
algorithms, such as request/reply or publish/subscribe. The algorithms handle
the issues like message routing, scaling up, handling partial failure etc.

~~~
derefr
Well, that's what I meant--technically, BSD sockets are themselves supposed to
be "a platform for delivering packaged and reusable distributed algorithms".

In a friendly, ideal environment (like the Internet of the 1980s), req/rep is
"just" TCP/SCTP unicast, pub/sub is "just" UDP multicast, and so on. Issues
like "message routing [and] scaling up" are supposed to be handled on the link
layer (i.e. using ARP, BGP, etc.); issues like handling partial failure, on
the transport layer (if IP stacks weren't stuck in the 70s, we'd have such a
thing as multicast TCP, and "send TCP packet to host set, wait for ACK from
all hosts, and retransmit to failed hosts" would simply be its obvious send()
system-call semantic.)

We've just screwed the "friendly, ideal environment' pooch by putting not-
very-intelligent NATs and firewalls everywhere. So we have to, sadly,
reimplement all these nice things on top of the few things we've kept working:
TCP and UDP unicast from clients to "business-leased" IPs with
open+addressable ports. The necessity of things like this, UPnP, and
TURN/STUN/ICE sadden me sometimes, they really do; they're 100% examples of an
inner-platform[1] created because software developers have gone on for so long
only touching OSI layer 7, and thinking about the rest of the infrastructure
as a black box, that now the application layer is the only place you _can_
solve problems, because the infrastructure will no longer transport your
packets if they do anything special below it.

So I say, please, look past your abstractions, programmers, and learn what
makes the Internet tick! It's worth it to understand what your packets are
doing--you might want a property that you can already get by asking something
different of a lower layer. Or, even if not, you might just become--as Bret
Victor says--very unsatisfied with the state of things. :)

Oh, and start here, if anywhere:
[http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html](http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html)

\---

[1] [http://en.wikipedia.org/wiki/Inner-
platform_effect](http://en.wikipedia.org/wiki/Inner-platform_effect)

~~~
rumcajz
You are oversimplifying it. REQ/REP is not the same as TCP, unlike TCP it
works with multiple requesters and multiple repliers. Same with other
patterns.

The main distinction between L4 and scalability layer is that L4 works between
2 nodes, scalability between arbitrary number of nodes.

There's the special case of UDP multicast, but the multicast itself feels more
like a hack rather than a real L4 protocol.

------
alexchamberlain
I don't buy the `it's better because it's C rather than C++`. For example,
"Number of memory allocations is drastically reduced as intrusive containers
are used instead of C++ STL containers." can be solved in 3 ways:

1\. use an intrusive container in C++.

2\. Just use a `vector`, faster in most cases, regardless of complexity.

3\. Read about and use allocators properly.

~~~
angersock
Or, the other valid solution:

1\. Write the whole thing in valid C89

2\. Prevent users from having to use Boost or the rest of the C++ nonsense

3\. Users can now write trivial language bindings and link at runtime easily.

4\. Profit.

~

C++ is a cool bro and all, but honestly the solutions you've listed make it
annoying to work with this code outside of C++ on a platform with subgood
template support.

Moving a simple utility library like ZMQ to C makes a great deal of sense.

~~~
easytiger
Who said anything about boost? afair zmq compiles with no boost dependencies

~~~
angersock
Read the thread--as per usual, the stock response to criticisms of C++ (here,
the difficulty with intrusive lists) is "USE MOAR BOOST!1111one".

Now, it's a very useful library, but it's also an amazingly tired refrain.

------
audidude
The author appears to care about performance. That is why I'm surprised to see
the choice of switching to level-triggered interrupts instead of edge-
triggered. This really defeats one of the most important performance
improvements of epoll().

See `man epoll' for an excellent explanation of the difference.

~~~
X-Istence
The problem is that embedding an edge triggered file descriptor into an event
loop that expects level triggered is a HUGE pain in the ass. I wrote two
articles on the matter:

[http://funcptr.net/2012/09/10/zeromq---edge-triggered-
notifi...](http://funcptr.net/2012/09/10/zeromq---edge-triggered-
notification/) [http://funcptr.net/2013/04/20/embedding-zeromq-in-the-
libev-...](http://funcptr.net/2013/04/20/embedding-zeromq-in-the-libev-event-
loop/)

------
gizzlon
Background, from wikipedia:

 _" iMatix CEO Pieter Hintjens registered the zeromq.org domain in May 2007
and started the ØMQ project together with Martin Sustrik, who was its
architect and lead developer through March 2012.

[..]

In March 2012, due to disagreements over control of the project, some of the
original developers forked ØMQ as Crossroads I/O. Martin Sustrik has started
nanomsg, a reboot of the ØMQ core library."_

------
Bjoern
Since the title mentions it.

The ØMQ library is licensed under the GNU Lesser General Public License. All
add-ons and examples are published under the GNU General Public License.

[http://zeromq.org/area:licensing](http://zeromq.org/area:licensing)

------
malkia
Things that are normally valid C++ code that would get you into trouble if you
are library (some of them you are static one):

\- Globally constructed objects, called before main(). Yes you might say,
don't use them, but once you are in C++ land you can't be safe that everything
else you've decided to use does not use them.

\- Throwing exceptions. This is mostly debugger pain. If you've used (again)
without knowledge underlying code/library that uses exceptions (and note, you
don't) then you might be inadvertently throwing them to the user.

\- RAII does not work with longjmp/setjmp. If you have callback mechanisms,
and your client does longjmp on you, this won't work with any RAII done
objects.

\- C++ runtime. You would be requiring this to be loaded, which might become
nuisance in situations where your main application never used C++, you wrote a
plugin/shared lib using C++ code, and now there is the requirement. Not a big
deal after all.

\- C++ runtime again but for MSVC (although it's for the C runtime too to a
point). Now while you can make code that is more independent from the "C"
runtime for MSVC, it's harder to maintain this for C++ - you might run into
issue where mixing dynamically (shared) libraries would have to use different
(and even more different through WinSxS) MSVCPxxx.DLL libs (C++ runtime). In
all fairness, this is more of a Microsoft mishap, and not caring about
backward binary compatibility, but hey it's there.

\- Most of all, no matter what you do - you can't expose the really useful C++
things - like stl or qt5 or something else's objects/vectors/strings/maps/etc.
Well that's obvious - off course you can, but then you are basically forcing
the person using your library to go that route. If your interface is in "C"
not only he longer has this pressure, he can use it easier through FFI in
plethora of other language/runtimes without writing wrappers, and minimizing
the number of precompiled binaries that he has to provide (no linking
statically to C++ runtime would turn out bad at the end).

\- Case in point - STL's ITERATOR_DEBUG_LEVEL in MSVC. By default it's set to
2, which means make extensive checks of every map<>, vector<>, etc. once they
are released. This severely slows down the debug. It's useful, but it's not
feature that can be turned on or off. Most of 3rd party libraries, are playing
safe and don't do anything special, e.g. they expect ITERATOR_DEBUG_LEVEL=2.
But in your production code you might want even your Debug versions to be
somewhat faster. Most of your day won't be spent tracking down map<>,
vector<>, but other things. As such you go ahead and compile
ITERATOR_DEBUG_LEVEL=0, and then .... well you can not link anymore directly
to the said 3rd party libs. You either have to precompile them, which might be
fine, until you are given Middleware that is not compiled this way. And in all
cases it's very important that all these libs use at the end just one runtime,
not several (as it happens when you have tons of MSVCRxxx.dll/MSVCPxxx.dll in
your running image).

\- And there is more. For example for statically build libraries, global
constructors might be lost, if there is no symbol referring to them used by
the main application. So you might lose some important initialization step.

\- etc., etc., etc.

~~~
krichman
I just saw someone use a globally constructed object to instantiate some
global variables and it seemed like a neat solution, but I don't really know
C++. What causes it to go wrong?

~~~
malkia
It depends on compiler / librarian / linker. I'm talking from experience here,
not from deep knowledge or understanding the problem.

So basically what happens, is that if you have a lone variable globally
instantiated, but no one refers to it, it might be stripped from the final
linking, and as such any constructors/destructors associated with it would not
be called.

Now if you are linking directly to the .obj file where the variable is
declared it most likely would be put into account, but if the .obj is in a
.lib (.a) then it might not.

Again, do some experiments, try it out.

There is another nastiness with globally initialized variables, that one
should not expet some kind of order when the things are initialized. We've had
code in the past (Dreamcast) where globally initialized variables, which were
assigned the values of other globally initialized variables and then some
operation on them would not always work: e.g. this at the top level:

float x = 10;

float y = x + 20;

It worked most of the time, and then if someone started moving variables to
other object files, it stopped.

Check the [http://yosefk.com/c++fqa/](http://yosefk.com/c++fqa/) for more
gotchas.

------
mordae
I am hoping for some kind of solution for the DoS problems present in ZeroMQ
(and probably in nanomsg as well) when used in an Internet facing application.
For example the possibility to subscribe to an unlimited number of prefixes in
the PUBSUB model leads to memory explosion on the server side.

~~~
rumcajz
Good point. Would you mind bringing it up on nanomsg mailing list?

------
clarkevans
I'm curious about the differences mentioned in the threading section, "In
nanomsg the objects are not tightly bound to particular threads and thus these
problems don't exist."

Does this mean that threading is still used/required, or could nanomsg use
non-blocking techniques?

~~~
rumcajz
Threading is required as there's processing going in the background. Not sure
what you mean by non-blocking techniques. Lock-free algorithms maybe? These
are used in ZeroMQ as well.

~~~
JulianMorrison
They probably mean async/evented techniques. For example, provide a function
to get a FD that select()s as readable when stuff needs background processing,
and a do_background_step() method to call, so it can be hooked into a single
threaded event loop.

~~~
rumcajz
Yuck. Why would anyone want that kind of stuff. nanomsg does all its
processing in the background threads, same way as TCP does. I doubt anyone
would ask to have to call do_background_processing_step() to make TCP work.

Also, that kind of thing compromises internal working of the library, if the
user doesn't call do_background_step() fast enough. Timers may be delayed that
way etc.

------
ska
How does this project relate to crossroads IO, if at all?

~~~
vasquez
I'd say crossroads IO is dead, and would start porting my stuff to nanomsg (or
ZMQ).

------
belorn
In one of the linked blogs, he implies that Windows OS uses MIT licensed
software.

Is this true? Do windows 7/8 have any lists of open source software projects
that they incorporated into the operative system? It sound as a really
interesting read if such was the case.

~~~
dalke
Windows included (and may still include) parts of BSD. The IPV4 networking
code, and some utilities like ftp.exe and rcp.exe, came from BSD.

~~~
pjmlp
This is no longer true, as the IP stack was rewritten in Vista.

[http://technet.microsoft.com/en-
us/network/bb545475.aspx](http://technet.microsoft.com/en-
us/network/bb545475.aspx)

~~~
belorn
Did the old IP stack have any MIT licensed software, or was it all BSD?

~~~
pjmlp
I think it was only partially BSD, but I don't have any documentation at hand.

There might be a reference to it on Windows Internals series.

------
saghul
Out of curiosity, was libuv ever considered as the underlying i/o library?
Looks like some problems at a quick glance (named pipes for instance) are
already solved there.

~~~
rumcajz
There are no 'scalability protocols' such as req/rep or pub/sub in libuv.

~~~
saghul
libuv would just be the transports, can't those be built on top of the TCP and
Pipe handles exposed by it?

------
krakensden
This is kind of exciting. I know that mongrel2, for instance, hit the REQ/XREQ
nonsense he fixed.

------
brunoqc
I saw a while ago that zmq could handle disconnections better:
[http://lucumr.pocoo.org/2012/6/26/disconnects-are-good-
for-y...](http://lucumr.pocoo.org/2012/6/26/disconnects-are-good-for-you/)

Does Nanomsg has the same problem?

~~~
rumcajz
The blog post was wirtten by Armin Ronacher. Here's today's tweet from Armin:
"Oh wow. Looks like nanomsg fixes all my problems I have with ZeroMQ."

~~~
brunoqc
Nice, thanks!

------
tel
I'm interested to see how the Nanomsg filesystem state model compares to the
ZMQ context state model. I'm a little bit worried about using Nanomsg in a
multithreaded environment.

~~~
rumcajz
The two are equivalent. The only difference is that in ZMQ the global state is
created upon zmq_init() call. In nanomsg it's created upon first nn_socket()
call.

~~~
tel
Is there a resource talking about thread safety in Nanomsg? I was writing a
Haskell client library a little while ago and couldn't quite figure out how to
treat threading, though I haven't had a chance to get back to it in a bit.

------
MetaCosm
The inproc fixes in nanonmsg are enough for me to start looking at it -- his
post explicitly hit on (and claims is fixed) the two most annoying parts of
inproc://

