

Dissecting Message Queues - swah
http://www.bravenewgeek.com/dissecting-message-queues/

======
codahale
You separate your load generation tools from your system under test to
eliminate confounding factors of shared resources like CPU, thread scheduler,
memory allocator, disk, etc. This benchmark doesn’t do that, which means it’s
impossible to distinguish between a queue system which is being saturated with
work and a queue system which is out-competing the load generators for CPU.
Also, it’s a laptop running OS X. Do you plan on fielding a queue system in a
DC built out with Macbooks? No? Then this benchmark might as well be on a
phone for all the inferential value it provides to people running Linux
servers.

A single producer and single consumer means zero contention for either for
most implementations. How well does this scale to your actual workload? What’s
the overhead of a producer or a consumer? What’s the saturation point or the
sustainable throughput according to the Universal Scalability Law? It’s
impossible to tell, since this is a data point of one, which means it can’t
distinguish between a system which has a mutex around accepting producer
connections and a purely lock-free system. And that’s a shame because wow
would those two systems have very different behaviors in production
environments w/ real workloads.

Finally, measuring the mean of the latency distribution is wrong. Latency is
never normally distributed, and if they recorded the standard deviation they’d
notice it’s several times larger than the mean. What matters with latency are
quantiles, ideally corrected for coordinated omission
([http://www.infoq.com/presentations/latency-
pitfalls](http://www.infoq.com/presentations/latency-pitfalls)).

This is not a benchmark, this is a diary entry.

~~~
hox
Not only that, but there is no attempt made to gather metrics based on similar
configurations / feature sets. The author specifically mentions that the AMQP
systems persist their messages to disk by default, and this was the
configuration used for the testing. How, then, are the "benchmarks" even
comparable to the ephemeral message queues that don't provide any sort of
persistence? Why wasn't persistence turned off to provide more comparable
tests?

And why were nanomgs / 0mq even included?

I despise articles like this. They only serve to clutter up useful
communication on such technologies.

~~~
tylertreat
Persistence was disabled. Nano and ZMQ are in their own group and aren't even
attempted to be compared to other groups.

------
kristaps
> What's interesting, however, is the disparity in the sender-to-receiver
> ratios. ZeroMQ is capable of sending over 5,000,000 messages per second but
> is only able to receive about 600,000/second.

IIRC the sent messages get stuffed in a buffer in the sender process. If the
sender process crashes it will take the already-sent-but-not-really messages
with it.

~~~
alexchamberlain
My understanding is that the "foreground" thread gives the message to a
background ZeroMQ thread, which then transmits them as fast as it can;
therefore, this number is probably somewhat misleading, but I'm not sure how
else you could measure it.

------
pyrophane
I might have missed this in the article, but were the brokered queues tested
in or close to their default configuration? ActiveMQ in particular needs to be
tweaked to get decent performance out of it in a production environment.

------
donatj
A little sad it doesn't cover gearman. Dead simple, easy to use, easy to get
going, super flexible.

------
quotemstr
No discussion of MSMQ?

~~~
easytiger
Almost no massively distributed applications are written on Windows, so i can
see how it would be overlooked.

