I've played with ZeroMQ on some small projects, and I've been quite impressed by it. Basically, ZeroMQ was designed to send millions of small, mostly-asynchronous messages at extremely high speeds. If you feed it messages very rapidly, it sends the first message immediately, and queues the rest in userspace until (IIRC) the network card is ready to transmit another packet. Then it sends all the queued messages as a batch. So you get both high throughput and low latency.
There's some built-in support for common topologies (client/server, publish/subscribe, pipeline), and if you're willing to use the undocumented XREQ/XREP socket types, you can build more exotic topologies.
Most of the value in ZeroMQ is the actual C++ implementation, so other languages generally just wrap the C++ code. The zeromq API is tiny.
I haven't used it on a big production project yet, but I hope to do that soon.
Well, the idea that simply 'queueing the rest in userspace' is OK illustrates a pretty disconcerting lack of understanding in writing networking code. This is not an acceptable solution in the real world.
I want to correct that assumption right now - it's one of those places that higher level networking libraries often fall down - because simply 'fire and forget' is a leaky abstraction.
Let's say you're streaming a large file over TCP to a client. Generally, disk reads are far faster than you can send data over the internet. If you naively keep sending data, you'll quickly fill up the OS buffers - and begin to consume memory as it's queued on the client side. In an extreme case, if you're sending a multi-GB file, you'll probably end up with most of the file sitting in RAM (assuming you don't run of ram and start thrashing). And for what? You really only need to load additional data in from the file as fast as you can send it in the socket.
In a real-life case, streaming multiple files to multiple clients, you'll quickly run out of RAM with even moderate size files.
Some high level libraries provide an event to let you know that the OS buffers have been emptied out. For example, Node.js has a drain event. But this has to be handled explicitly, and isn't really highlighted in the docs. But if you write your code naively, you will definitely run into problems.
While it's great that many of these high level networking libraries have tried to abstract away many of the complexities of network programming, they're also a trap for the unwary. Just remember that they aren't a silver bullet that let you handle sending data over the internet as a simple function call.
Have you checked out the ZeroMQ documentation on flow control and data rate limiting? I can't speak for anyone else's use case, but there's a nice selection of blocking primitives and data rate limits that have proven sufficient for the smaller programs I've written.
It's hard for me to tell if your criticisms are based on real-world experience with the library, a review of the available documentation, or just my description of a single feature intended to reduce the cost traversing the userspace/kernel barrier.
I'm definitely interested in real-world experience with ZeroMQ and any problems you've encountered. We may begin some larger-scale testing soon, and would love to know about any problems.
Instead of rate-limiting, there are two other options that solve the GP's concerns:
the high water mark which controls the maximum queue length and the swap size,
which saves messages to disk when the queue is full.
If you are streaming a large file over TCP, ZeroMQ, with its emphasis on small messages sent and received apparently atomically, would be a stunningly poor choice.
It's also wholly unsuitable as a floor wax or a dessert topping. :)
Does your criticism also apply to its intended use case? At a glance, it doesn't strike me as an unreasonable design choice, but I haven't thought about it deeply.
Are you intending to have many of these transfers happening in parallel, or are you sending a single file and hoping to saturate your link? Do you have full control of both ends, or only the server? Are you streaming or is the whole file always transferred? Do you need encryption?
For simultaneous large files with no special needs, my instinct would be to just choose any lightweight HTTP server that supports sendfile(). Basically, pass it off to the kernel, and let it handle it. But maybe there are more subtle keys to good large file performance: http://www.psc.edu/networking/projects/tcptune/
0MQ is built on good old principles the Internet stack (TCP, IP) is based upon. Thus is has tx/rx buffers. It doesn't even try to do fire-and-forget, because -- as you correctly say -- it's an leaky abstraction. Instead it aims for end-to-end reliability. And yes, moving it into kernel would make it somehow more reliable. Watch for my talk at Linux Kongress (eeptember 23rd) which deals with moving the functionality into kernel, standardisation and integration with Internet stack.
0MQ has what is called a high-water mark on (outgoing) queues. You can then decide whether to block if the queue gets full, or poll the socket to know when it's ready for output. This is also how TCP works. 0MQ abstracts the complexities of the network that you don't want to see, but it exposes those parts you still need to. It would make a very nice transport for file transfer, and I'll probably add such an example to the 0MQ Guide.
xrep/xreq really isn't well documented. I'm writing a learn by example series for ZeroMQ in Ruby, that already includes an XREP example. I'm planning on adding in XREQ this weekend.
There's some built-in support for common topologies (client/server, publish/subscribe, pipeline), and if you're willing to use the undocumented XREQ/XREP socket types, you can build more exotic topologies.
Most of the value in ZeroMQ is the actual C++ implementation, so other languages generally just wrap the C++ code. The zeromq API is tiny.
I haven't used it on a big production project yet, but I hope to do that soon.