Watch the first video by Ian Barber. He's an excellent presenter.
The guide ( http://zguide.zeromq.org/page:all ) is very lengthy and comprehensive, but the "hello world" example and the "Divide and Conquer" example (fan out jobs, work, fan in results) will get you up and running and start showing you a sliver of the amazing power of 0MQ.
They've even got a lot of their examples in many languages:
C++ | C# | Clojure | CL | Erlang | F# | Haskell | Haxe | Java | Lua | Node.js | Perl | PHP | Python | Ruby | Scala | Ada | Basic | Go | Objective-C | ooc
ZeroMQ: Modern & Fast Networking Stack, by Ilya Grigorik
ZeroMQ an Introduction, by Nicholas Piël
Advanced Network Architectures with ZeroMQ, by Zed Shaw
Does anyone know of a good, detailed explanation of how 0MQ is implemented?
A lot of the behavior is on a per-protocol and socket type basis, which is well documented here: http://api.zeromq.org/2-1:zmq-socket
And you're right that some explanation of implementation is needed. I spent a few hours digging through the code today and it's not easy to read.
ZeroMQ and XMPP are not really competitors.
The "Zero" in ZeroMQ means there is no broker (server) between the publisher and the subscribers. Instead what happens is that each process runs a separate thread which deals exclusively with ZeroMQ messaging.
Another effect of having no broker is there is no disk persistence at all, so unless you take your own steps you will lose messages if any component crashes.
3.x removes the asserts so you can't segfault remotely. 3.x also transparently drops messages on the floor destined for unroutable identities. This silent dropping sounds annoying, but you can work around it via a well-designed application protocol. Remember, 0MQ is effectively a transport layer, not a full-blown messaging system.
Would it be applicable to large-scale optimization problems, e.g. in context of machine learning?
But it stops there (and that is the beauty of it). It doesn't prescribe or make assumptions about the types of messages that are sent. Those are just byte blobs as far it it is concerned. You can choose to represent your messages as json, protobuf, xml, raw video frames.
So can you use it for HPC? It is hard to tell, sure, it is like asking if hammer can be used to be fix a car. Yeah, it can, in some cases, but it is just a good versatile tool, that can be used for more than that.
I am currently using it to transfer model parameters learned on machine #1 to machine #2 where real time prediction takes place. The predictions are then sent out via 0mq to a robot controller.
The great thing of building systems with 0mq is that you just add another component without having to think too much about 'but then I will have to implement a protocol for it and HTTP will probably to slow and the request/reply pattern doesn't really fit there blabla'.
I've only been using 0MQ as a library and AMQP in the form of RabbitMQ - so it can be quite the opposite.
edit: and same for if you run your own server or rent a server, obviously
If you're only using ZeroMQ in tcp or inproc mode, it is nice, but in a pub/sub scenario you are wasting a lot of bandwidth and CPU time.