

Evaluating persistent, replicated message queues - adamw1pl
http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/

======
twic
It would be nice to see the strongest of the various Apache contenders
represented here too. Once upon a time, there was just ActiveMQ to worry
about:

[http://activemq.apache.org/](http://activemq.apache.org/)

It's long been very popular, and has all the flexibility and enterprisey
goodness you could want, but has not had a reputation for speed. However, i
believe there was a release a year or two ago that significantly improved
performance. I vaguely remember this being based on an infusion of technology
from the Qpid project, which itself has a broker:

[http://qpid.apache.org/components/java-
broker/index.html](http://qpid.apache.org/components/java-broker/index.html)

Or maybe that was what led to Apollo, which is the faster, more reliable
successor to ActiveMQ, but which hasn't, er, succeeded ActiveMQ:

[http://activemq.apache.org/apollo/](http://activemq.apache.org/apollo/)

Or maybe i just imagined the link to Qpid.

I'm fairly hazy on how these bits of software relate, and i can't promise you
there aren't more somewhere in the vast Apache mansion. Even figuring that out
would be useful, never mind the benchmarks!

~~~
adamw1pl
I'm also not sure how the above projects relate. As far as the scope of the
blog goes, Apollo seems to be ruled out as it is not replicated.

ActiveMQ does have replication, but they don't really sound convincing in the
docs that it's first-class ;)

------
srean
I am not very familiar with messaging or message queues, so wanted to seek out
some advice before spending time reinventing some well known wheel poorly.

I would like to have a broker that sits between an application and several
endpoint services to which the application makes requests via the broker.
These endpoint services have rules, typically throttling rules (i) you cannot
submit more than x number of request per second (ii) you cannot queue up more
than y number of requests at our end (iii) you must put a delay of z seconds
after we have responded to a request of yours, etc., etc.

Is there a defacto standard open source software for this situation, that
would take a config file describing the policies to follow for an end point.
My requirement is pretty toy'ish, small to medium scale only. Just taking a
look at Redis docs (am new to that too) it seems it will not be that hard to
write out a simple solution. Would that be the way to go ? Or is there
something out there that already handles this.

~~~
adamw1pl
HornetQ has flow control
([http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-
manual/h...](http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-
manual/html_single/index.html#flow-control)). Not sure about e.g. Rabbit.

However I'm not aware of a standard solution in that space.

Take a look at Reactive Streams ([http://www.reactive-
streams.org/](http://www.reactive-streams.org/)) as well.

------
sargun
What this article fails to talk about are primarily the failure scenarios of
the queues under partitions. Does the queue become write-unavailable, read-
unavailable, inconsistent, or otherwise during network partition? Can the
queue remain partially available during network partition?

These are the situations, in a scaling, production system that really matter.

~~~
adamw1pl
These are very good questions, however for a different post. Here I was mainly
interested in the throughput.

Jepsen post provide some insights (into Mongo and Rabbit). With Hornet you can
easily end up with two primaries. SQS is untestable in that respect :).

------
baq
required reading for anyone using distributed systems:
[http://aphyr.com/tags/Jepsen](http://aphyr.com/tags/Jepsen)

in particular:

[http://aphyr.com/posts/284-call-me-maybe-
mongodb](http://aphyr.com/posts/284-call-me-maybe-mongodb)

[http://aphyr.com/posts/315-call-me-maybe-
rabbitmq](http://aphyr.com/posts/315-call-me-maybe-rabbitmq)

~~~
adamw1pl
Agreed, a must-read :). Bottom line is all queues can loose messages (we don't
know much about SQS in fact).

------
whalesalad
Would love to see Kafka represented here.

~~~
adamw1pl
Yes, my bad, should be there :)

Stay tuned, I'm out soon till end of next week, but will definitely add it.

