A single producer and single consumer means zero contention for either for most implementations. How well does this scale to your actual workload? What’s the overhead of a producer or a consumer? What’s the saturation point or the sustainable throughput according to the Universal Scalability Law? It’s impossible to tell, since this is a data point of one, which means it can’t distinguish between a system which has a mutex around accepting producer connections and a purely lock-free system. And that’s a shame because wow would those two systems have very different behaviors in production environments w/ real workloads.
Finally, measuring the mean of the latency distribution is wrong. Latency is never normally distributed, and if they recorded the standard deviation they’d notice it’s several times larger than the mean. What matters with latency are quantiles, ideally corrected for coordinated omission (http://www.infoq.com/presentations/latency-pitfalls).
This is not a benchmark, this is a diary entry.
And why were nanomgs / 0mq even included?
I despise articles like this. They only serve to clutter up useful communication on such technologies.
We're attempting to optimize this aspect of our stack currently, and I'm sure many others face very similar challenges right now. It's proven to be quite difficult & time-consuming to accurately measure this stuff -- any insight into more accurate/reasonably realistic benchmarks of this type of MQ software would be awesome. :-)
Some of the stuff is difficult and time-consuming because "messaging" is generic enough to be configured and used differently by different users.
Obviously you can cut away some of the choicesright of the bat if you are worried about support for some OS (Like you have to ship on HP UX), or you need to have durability and acknowledgement and high availability, or you want a project with a certain level of maturity and stability and so on. That cuts the number of systems to test.
Then of course there are things like, well how do they handle concurrency. Just because a single producer and single consumer can do 500K messages per second (which maybe a small benchmark on a co-workers laptop will show), doesn't mean that the whole thing won't blow up and crash in a burning mess if there are 1000 consumers and producers.
IIRC the sent messages get stuffed in a buffer in the sender process. If the sender process crashes it will take the already-sent-but-not-really messages with it.