
A quick message queue benchmark: ActiveMQ, RabbitMQ, HornetQ, QPID, Apollo - liotier
http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
======
rryan
I suspect this benchmark is actually just testing the difference between STOMP
and AMQP. RabbitMQ with STOMP is in the same regime as ActiveMQ / Apollo while
RabbitMQ with AMQP is always 3x-5x faster than _itself_ and others with STOMP.

QPID is the only other AMQP competitor and it seems to have performance issues
with persistence but the transient performance is always in the same regime as
AMQP RabbitMQ.

STOMP is a UTF-8 text protocol while AMQP is a binary protocol. This means
that every message you send using STOMP requires encoding it in a safe-for-
text format (e.g. base64 encoding or similar. I'm not intimately familiar with
STOMP).

Going on the hypothesis that a STOMP message parser is much slower than an
AMQP parser, this would explain why RabbitMQ AMQP does not perform well in the
200 message case while it trounces in the 20k and 200k message case. In the
20k/200k message case the benchmark is mostly looking at time to decode 200k
text messages versus 200k binary messages.

It's so exceedingly hard to do a benchmark and measure the thing you think you
are measuring.

Also, I agree with others that the minimal difference between persistent and
transient setups is very non-intuitive. The author left out an important
detail of whether he was using an SSD or not. Otherwise, I suspect an error in
the measurement setup.

------
noselasd
Please keep in mind:

\- All these products have _a lot_ of knobs to set. Performance depends on
these knobs. It depends on whether you do batch sends(which some can do),
whether you do transactions, whether you chose at-least-once, at-most-once
etc. delivery, queue prefetch sizes, I/O worker size, and numerous other
settings.

It'd also be interresting to see qpid-cpp in there as well, not just the Java
version.

~~~
maybenot
This is a test of the default configurations of various brokers, NOT the
brokers themselves.

 _Apart from declaring the testing queues in some brokers’ configuration and
the persistence settings, all brokers were running with their default
configuration out of the box (no tuning made)._

He also casually mentions that ZeroMQ is using an in-memory configuration.

 _A home-made ZeroMQ 2.2.0 broker, working in memory only (no persistence)._

The default configuration of AMQ persists messages to journal on disk. So
apples to oranges?

Why do people keep writing "benchmark" blog posts like this?

~~~
jdcryans
My thoughts exactly, this is useless. Benchmarking is something everyone can
do, so people do it, but it doesn't provide any real value unless you put an
insane amount of time into it, and even then you'll get a lot of hate mail :)

For example, look at the "Yahoo! Cloud System Benchmark" AKA YCSB paper. Those
researchers spent a lot of time designing the benchmarks and properly
configuring each database that they were testing. They even dedicated a
researcher per database to sit down with the developers to review the
configurations and the test runs. I was part of this process for the Apache
HBase as a dev. In the end, everyone was still critical of the results once
they saw the graphs comparing the DBs.

But, I'm still glad they did it as I use YCSB as one of my tools to benchmark
HBase.

------
mootothemax
Interesting results! In my testing, I came to a different conclusion.

Background: PHP and STOMP, with loads of several million messages per hour,
ranging in size from 2-20Kb. Multiple processes enqueuing, and many more
dequeuing. Tests performed about this time last year.

Out of ActiveMQ, RabbitMQ and Apollo:

ActiveMQ crashed _constantly_ under load.

RabbitMQ could not enqueue/dequeue fast enough.

Apollo blew them all out of the water _and_ , long-term, has proven extremely
stable.

I must admit that haven't spent hours performance tuning. The results shown
here, however, make me think I should go back and re-evaluate.

~~~
mbell
> ActiveMQ crashed constantly under load.

That has been my experience as well, to this day I have an ActiveMQ broker for
which the JVM just stops responding, its not the broker that goes
unresponsive, its the entire JVM, with no error, no log, no nothing, it just
stops replying to the service wrapper and sits at 100% cpu usage. The wrapper
kills that JVM and restarts it once every couple days because this happens.

Oh and this was fun: yesterday I went to check on the broker as it was running
a bit slow, the DLQ had ~50,000 messages in it that were not there last week,
all were timestamped to the same second in ~July 2012...9 months ago...

I'll likely never use ActiveMQ again.

~~~
maybenot
AMQ is sensitive to proper memory configuration. Typically what will happen is
the memory configuration in the activemq.xml doesn't line up with the JVM heap
and this causes garbage collection thrashing. It reaches a point where the CPU
spikes and the monitor process zaps the JVM process, which causes a restart.

If everything is tuned and lined up properly this won't happen. I've used AMQ
in high-volume production systems for years without this problem.

~~~
mbell
I've played with this and am not getting anywhere near the -Xmx set for the
JVM and producer flow control isn't kicking it. Load on this broker isn't very
high either, 5k to 10k messages/day, all messages << 1KB.

------
rabbitmq
Sidebar --- anyone interested in running tests with RabbitMQ should check out
these posts as a starting point.

[http://www.rabbitmq.com/blog/2012/04/17/rabbitmq-
performance...](http://www.rabbitmq.com/blog/2012/04/17/rabbitmq-performance-
measurements-part-1/)

[http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-
performance...](http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-
measurements-part-2/)

Please note that:

1\. the results are from last year, and so not 100% representative of RabbitMQ
3.x which was released after these blogs were posted

2\. what actually matters with message queue performance is stability and
scalability over time .. testing this is extremely hard without making over-
specific assumptions

------
adaml_623
I appreciate that some effort went into coding this benchmark but I don't
understand no time was spent trying to optimise each MQ. You could even email
the developers and ask for help since you're going to be publishing this as a
comparative benchmark.

------
verelo
I really dislike graphs that leave labels off. When presenting results its
nice to have a clear understanding of whats being displayed, the missing Y
axis on these graphs is not helping.

~~~
VikingCoder
"And now, the results (processing time measured in seconds: the lower the
better)."

------
knodi
Anyone here use NSQ in production (<https://github.com/bitly/nsq>). If so what
are your thoughts in comparison to the article?

~~~
mh-
we do; rather happy with it. do you have any specific questions about it? the
article's approach is.. strange, so I'm not sure how I can meaningfully
contrast from a high-level.

 _(disclosure: i've contributed to nsq, after we began using it in
production)_

~~~
daemon13
Any gotchas moving from RabbitMQ to nsq?

Also, are you missing any features in nsq?

~~~
mh-
didn't run into any gotchas, but I tested things very thoroughly before going
into production.

to the contrary- deploying static binaries (instead of the erlang environment)
simplified things nicely, in my opinion. to be fair, i (personally) don't have
the requisite experience tuning BEAM which probably biases my preference.

while relatively high volume, our usage of RabbitMQ was straightforward and
covered by the functionality offered in NSQ.

like most, we were using AMQP.. so the switch to NSQ's concise wire protocol
(and the associated reduction in pkt/s) saved us a lot of pain given the
highly-variable performance we see in the EC2 network.

------
DomBlack
I'm interested to see ZMQ being compared to the traditional brokers as I've
been considering swapping from ActiveMQ. Although not surprised at it's
results. It very much re-enforces the ideal of keeping software simple and
barebones, rather than bloating it with stuff most people never use.

Mainly because I'm finding the latency from ActiveMQ is starting to affect my
overall system latency. RabbitMQ was the next broker on my list to test, but
ZMQ makes more sense (if you don't mind writing the broker part).

Background; I'm currently running a system which at peak runs with about 15
million messages per hour using ActiveMQ, with several producers and consumers
on the same topic. Apart from speed, not had any issues with it.

~~~
FooBarWidget
I would not use ZeroMQ for anything but the most non-critical queues. ZeroMQ
is not persistent: if your server crashes you lose all your queue contents.

My current favorite is RabbitMQ. It has improved steadily over the years,
performs pretty well and very easy to setup.

~~~
sneak
ZeroMQ is a messaging library only.

There is nothing stopping you from using ZeroMQ to build a daemon that
implements a persistent queue— in fact, I have, several times. It's very
simple.

~~~
FooBarWidget
And then you've made your own broker. At that point why not use an existing,
more mature broker?

~~~
sneak
The zero in ZeroMQ is because brokerlessness is a strength in some aspects.

A broker can be an SPOF, for instance. 0MQ makes it easy to do shared-nothing.

~~~
FooBarWidget
I know what ZeroMQ does. But you argue that a one can easily write a broker on
top of ZeroMQ that provides persistency, but that invalidates your biggest
reason of using ZeroMQ, namely brokerlessness.

Besides SPOF is not an argument. All serious brokers have supported clustering
and failover for quite a while, _and_ they support persistency. I don't see a
good reason why you may want to write a ZeroMQ-based broker instead of just
using RabbitMQ.

------
alberth
>>"Except for big messages, RabbitMQ seems to be the best bet as it
outperforms others by a factor of 3."

RabbitMQ? Does the OP mean to say ZeroMQ?

ZeroMQ leads every benchmark in the blog.

~~~
Aqua_Geek
True, but it's in-memory only (there was no persistence with ZeroMQ in the
given setup).

~~~
alberth
Then why even test ZeroMQ then?

If you plan to setup a benchmark test, and then decide to throw out the
results (winner/ZeroMQ) because it doesn't produce the desired outcome you
wanted ... then this benchmark is a joke to begin with.

~~~
dikei
He did say _ZeroMQ broker outperforms all others. This means that unless you
have a need for complex broker features, ZeroMQ is a perfect message
dispatcher among processes._

------
piokoch
I don't understand the point of making benchmark of these message brokers in
the default configuration (usually tailored for ease of development, not
performance). Who is using message brokers in the default configuration in
real deployments?

------
kodablah
I would like to see Kestrel (and Darner) in this benchmark. My personal
experience with Darner is that it handles extreme loads. Granted, the MQ is
not distributed and it is difficult to do "topics" in the JMS sense.

------
don_draper
When is a message queue required? And when is a pub/sub setup with Redis good
enough? I'm sure it's apples to oranges, but there must be a few places with
ActiveMQ installed where it wasn't really needed.

------
lucian1900
I'm a bit sceptical of this, as RabbitMQ performance is very similar between
transient and persistent. In my own tests, they were up to an order of
magnitude different, even on an SSD.

------
mvzink
I would rather see number of processes varied.

------
JimmaDaRustla
What, no ZeroMQ? (Page won't load)

