

Ask HN: How to choose a message queue? - porker

Why do we have 101 different message queues?<p>How can they be grouped? I see different message protocols being used; some are persistent. Are there other, more fundamental differences which make some suitable for microservices and others for job queues?<p>The comparisons I&#x27;ve found consider 4-5 of the queues and pit them against each other for speed - which while important isn&#x27;t everything.<p>Can you help with a higher-level overview and how to choose a suitable queue?
======
spotman
Choosing a message queue can certainly be tricky.

Not a complete list, but you should consider:

a.) do you need persistence? some queues offer this, some do not. many systems
can be configured in either mode. you might need persistence if the messages
your processing are vital and shouldn't be missed in the case of a broker
restart, for example. however if the messages are very volatile, then maybe
persistence is overkill. Or possibly, the architecture your working with
handles message loss somewhere else, so its not the message queues
responsibility to worry about loss.

b.) do you need fault tolerance? some queues off this, some do not. just like
persistence, its often something you can enable or disable. and like
persistence, often queues can be setup so that while they might not be fault
tolerant, the application that employs them is designed to work around queues
going down.

c.) do you need high availability? fault tolerance and high availability are
separate things. fault tolerance might just be a master falling back to a
slave. high availability means this happening and there being no hiccup in
service.

d.) performance. its something to consider, but your correct in thinking that
this is not everything. but sometimes its worth considering. in one experiment
I ran, Amazon SQS was about 40x slower than rabbitmq. This may or may not be
worth the trade off for you, as products like SQS relieve the developer and
operations a lot of responsibility, but if 10ms average queue time (or less)
is important, forget SQS.

e.) client libraries. some queues have client libraries for every language in
the sun. some only offer support for one. you would want to consider who and
what will need to talk to the message queue, and if this is going to be made a
nightmare if your using something very exotic or custom.

f.) creature comforts. some things come with nice rest api's, some come with
web ui's, some come with only command line interfaces, and some come with none
of that. Often things like ZeroMQ, Nanomsg, are put into these blogs and
things comparing message queues. They really are not message queues in the
traditional sense. They can only replace one after you've probably spent a
considerable amount of time building tooling. For example, in zeromq there is
no way to ask 'how many messages are pending in this queue' without writing
that functionality yourself.

g.) queues-as-a-service. I mentioned Amazon SQS in point d, but they are
something to consider if you don't want to run it yourself. Running a message
queue can be easy, but running a message queue that is fault tolerant, highly
available, deals with message loss/deliverability, and scales as your workload
scales, can be an art form.

In summary, you should do your own research based on these needs and others I
likely forgot. Sometimes its easiest to just pick a simple one, and then once
you experience its shortcomings and learn more about your queue needs, the
upgrade path will be much less mysterious.

~~~
hkarthik
Excellent response. Only thing I would add is this:

h) compatibility. Make sure (and test!) the compatibility of the queue with
your application stack. Even though a queue may advertise support for a
certain stack, the libraries will range in quality and every queue has a
"first class citizen" in terms of supported stack. If you're going to take a
chance on an off-the-beaten path combination of application stack and queue,
accept that you may need to extend or even fork the client library to make
sure it fits your needs. Queues vary in complexity, and often a lot of
complexity gets pushed to the client.

------
jgill
You need to really analyze your requirements. Why do you want/need a message
queue and not a database? Do you need to support message durability,
clustering of nodes, nodes in different geographical regions, topical
subscriptions, what kind of routing concerns, technology support (do you
want/need to write your own client to interface with the message queue?
hopfully not...if you're using a particular language you just might though),
protocol support (do you need amqp support), etc.?

Personally I like RabbitMQ.

------
s3b
Have a look at [http://www.bravenewgeek.com/dissecting-message-
queues/](http://www.bravenewgeek.com/dissecting-message-queues/)

------
shaunpud
[http://queues.io](http://queues.io)

~~~
porker
queues.io is a good list of message queues, but it's not a _useful_ list. It
says nothing about them, how they work, the differences or what they're suited
for.

For comparison, this is a useful list for getting the feel of a technology
field: [http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-
redis](http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis)

