
ActiveMQ: Not ready for prime time - sorenbs
http://goodstuff.im/activemq-not-ready-for-prime-time
======
lkesteloot
I used ActiveMQ in a large production environment for two years. It's by a
large margin the worst piece of software I've used professionally. You
wouldn't believe the number of very serious problems we had with it (including
some mentioned here, like negative queue lengths, but mostly broker crashes,
missing or wrong documentation, outright broken features, serious threading
issues, and poor performance). If anyone's not convinced, just peek at the
source code. It's frightening.

We ended up writing a new queuing system from scratch in a month, and in its
first week it was already more stable, performant, and bug-free than ActiveMQ.

~~~
gaius
The thing we found is that when it crashed under load, it could take _over_ 24
hours to consistency check and rebuild its queues on-disk. Insane for a
highly-available environment.

This is why you should NEVER let an architect choose your software - ALWAYS
the sysadmin who will actually be on-call to support it should have the final
say.

~~~
CrLf
"ALWAYS the sysadmin who will actually be on-call to support it should have
the final say"

Ah, if only that happened in the real world...

------
ShabbyDoo
I see lots of questions here about which queuing implementations one ought to
consider. Some things to think about:

1\. Do you need transactionality? What if your consumer experiences an error
after it takes a message off the queue but before it finishes up processing
it? Is it ok for that message to disappear forever? The common use case to
consider is a consumer which takes a message from a queue, does some
processing, and updates a database. To avoid distributed transactions, people
often write these consumers so that they check first to see if a message has
already been processed. This way, one simply ensures that his DB transaction
commits before committing to the consumption of the message.

2\. What level of message durability is required? Is it ok for all enqueued
messages to disappear when a power cord is inadvertently pulled? I used
ActiveMQ to populate a user behavior data warehouse. As users did stuff on a
website, the application servers would enqueue observations (went to the
product detail page, etc.) and these observations would eventually wind-up in
a big Oracle DB. A delay of a day or so didn't matter, but we didn't want to
lose more than a fraction of a percent of our observations. So, the queue was
in effect a temporary system of record, and we had to allow for reboots, power
outages, etc.

3\. High Availability. Many modern queueing implementations can be deployed in
redundant, scalable "meshes", ActiveMQ included. I haven't kept up on the
feature sets of RabbitMQ, JBoss's offering, etc. or I would comment more here.

There are obviously a bunch of other considerations, but these are often the
ones which are ignored when comparing ZeroMQ to ActiveMQ, etc. There's no
right or wrong answer. "Lighter" implementations are often appropriate for
messages where, if the shit hits the fan with your environment, the messages
are useless anyway. IIRC, eBay builds pages by firing off a bunch of async
requests for page parts, waits a maximum of N milliseconds, and then renders
the page based on which pieces made it back on time. There's no value in
persistence or transactionality, but the mesh sure better scale up.

------
rbranson
It's absolutely incredible that it can take so much effort just to make it
run. ActiveMQ seems shiny and interesting on the outside, but the prolific
number of war stories took it off my list.

In brilliant contrast, RabbitMQ is it-just-works software. It's a piece of
infrastructure that works so well it practically just fades into the
background. I couldn't imagine having to worry about a service in which the
chief goal is to help alleviate reliability concerns.

~~~
Uchikoma
The Reddit guys think differently: "[...] (rabbitmq) died, which added about
an hour to the downtime. It dies like this pretty often at 2am or at other
especially bad times. Usually it doesn't cause any data-loss, just sleep-loss
(its queues are persisted and the apps just build up their own queues until it
comes back up), but in this case it decided to crash in a way that corrupted
its database of persisted queues beyond repair. rabbitmq accounts for the only
unrecoverable data-loss incurred, which was about 400 votes. [...]
Coincidentally, rabbitmq crashed twice more that day and a few more times into
the weekend. [...] Things have improved thus far, but replacing rabbitmq is at
the top end of our extremely long list of things to do."

~~~
rbranson
"Crashed"... I'm glad they're using such specific terms. I give them a lot of
slack because they run that shop with a skeleton crew, but they sure do run
into a lot of issues with perfectly good software, have Twitter levels of
performance & availability, and make some very odd technical decisions.

------
kitplummer
Yikes. If you're smart, you'll do your diligence and not rely on one person's
experience with ActiveMQ. Remember, it is an Apache project - so if something
aint there, feel free to fill it in (re: documentation).

I can't believe the ActiveMQ/FuseSource guys weren't willing to work it out
with David either. Perhaps there's a bit of open source/commercial software
wrangling behind this story. Won't be the first/last time.

~~~
rajdavies
I'm not sure who David Pollak talked to at FuseSource - nobody on the ActiveMQ
team there know him - and the execs are scratching their heads too.

------
mgkimsal
So, what might you recommend instead? I'm using rabbitmq on a couple of
projects, and have been a bit underwhelmed by the lack of management tools
that come with it. I figured something as basic as getting a list of what's in
a queue and being able to then remove that item from a queue would not require
me to write custom code, but I seem to have been wrong.

~~~
timclark
I believe the RabbitMQ management plugin is what you need, provides HTML user
interface and HTML API.

The BQL plugin, which is now sadly unsupported by the Rabbits, used to be a
nice command line interface, it is a shame it is no longer supported.

~~~
AmrMostafa
The RabbitMQ management plugin _does_ provide an excellent command line
interface. You download it along with its docs right from the management UI
(/cli).

Btw, I'm using RabbitMQ, and I love it. My needs do not include high load or
high availability though so I can't speak for that. So what's nice about it?
AMQP (you automatically get lots of tools, docs, "expectations", etc), very
friendly and active mailing lists, small & clean code base, small footprint,
simple, active (more features are always being added).

What I don't like about RabbitMQ? While it's FLOSS inside out, its development
isn't exactly a "community" work. For example, I can't report an issue, attach
a patch, and receive a reviewer/committer feedback about it and possibly get
it in. In fact, I can't even report an issue into their issue tracking system
-- I just have the mailing list. That said, I believe they said they are going
to fix that part "soon".

~~~
rabbitmq
We have always accepted server patches via the mailing list:
[https://lists.rabbitmq.com/cgi-
bin/mailman/listinfo/rabbitmq...](https://lists.rabbitmq.com/cgi-
bin/mailman/listinfo/rabbitmq-discuss)

But it's true that the community has been much more involved in clients like
Pika, for example.

And yes we _are_ planning to open up the bug tracker.

A piece of advice for anyone doing an open source project - start with an open
tracker, because opening up a previously closed tracker is a royal pain in the
butt.

~~~
dpw
And while we don't use github internally, people can and do submit issues and
pull requests there:

<https://github.com/rabbitmq>

(N.B. we need contributors to sign a contributor agreement.)

David (rabbiteer)

------
adamgravitis
The thing I've found about message queues is that it's pretty unimportant what
platform they're based on. I used to be tempted to go with ActiveMQ "because
it runs on the JVM", but that turns out to be pretty irrelevant.

No matter what your queue provider, you're most likely just using TCP sockets
anyway, so don't think you need to be using Erlang just to take advantage of
RabbitMQ.

Yes, this means the MQ can basically be thought of as a black box. But that's
also kind of the point.

~~~
marshray
What I've seen with RabbitMQ is that the memory usage can grow and grow. If it
runs out of either committed RAM or address space, it crashes. I suspect this
may be a feature of the open source Erlang VM that's improved in the pay-for-
support version. Even though RabbitMQ pays the performance cost of bouncing
everything off the disk, it's not quite recoverable from that type of crash.

I'm sure my information is old, perhaps these things are improved in newer
versions. Still, the behavior of the Erlang VM itself was relevant.

As a C++ programmer, I wouldn't feel comfortable abstracting away memory
management for this type of server which is both so memory intensive and
requires high reliability.

But of course many people swear by highly reliable JVM based stuff, so perhaps
I'm wrong about it. :-)

~~~
rbranson
Actually Erlang/OTP provides exactly the type of abstraction layer that's
extremely useful for RabbitMQ's use case. It's specifically designed for
consistent latency, predictable memory usage, and resilience under failure
conditions. You know, basically the things that make an MQ worth using.

It was true that previously RabbitMQ could run out of memory and be unable to
take more messages, but that changed in 2.0 and it will now swap messages to
disk. Of course, then you pay the consequences of slow disks and can always
run out of disk space, but nothing is invincible.

------
nonrecursive
Does anyone have recommendations for resources on messaging/queueing basics?
I'm look for something on the basic patterns (pubsub, broadcast... ?) and
basic considerations to be aware of.

~~~
andrewcooke
<http://www.eaipatterns.com/> perhaps?

~~~
vailripper
This is what I used to get up to speed on integration patterns. The book that
the site is based on is very very valuable as well.

------
ibejoeb
> It's not correctly configured out of the box.

You mean there's a _correct_ configuration? Also, Tomcat, Jetty, PostgreSQL,
and Ngnix don't "install correctly" out of the box as described here. Tomcat
clusters set themselves up? Postgres? Hell, I think default shared buffers
allocation is 32 MB...

> Anything that streams bytes

I'm not sure what the author is trying to do. Doesn't sound like queuing to
me...

The documentation is terrible, so I'll give that up. It is unfortunate that
there is so much experimentation involved in setting it up. ActiveMQ is
immensely versatile and configurable, and it's sort of necessarily complex to
get it perfect for your task. It's not a turn-key software; it's a systems
architecture component.

~~~
brown9-2
The part about Tomcat certainly isn't true, unless the author is using a
different distribution than what you get from the project's website. The
Tomcat startup scripts do not set any heap size paramters; until recently the
default Sun JVM behavior was to set a max heap size of _only_ 64MB. This logic
changed a few versions ago to use a fractional amount of the total memory size
on the machine, but for dedicated hosts this is still rarely the setup you
want.

Perhaps the author is using a packaged version of Tomcat from his OS
distributor.

~~~
ibejoeb
Yeah, I don't know what else it could be. It's up to the distribution
maintainers then. It's a straw man. ActiveMQ does just fine. I have a very
high volume implementation that I haven't touched in almost 2 years.

That it's not production ready is pretty big claim. I don't want _my_ clients
reading this and getting all jittery on me.

------
tvaillancourt
We use ActiveMQ in production. I couldn't agree more that it is not ready for
prime time. We've had a pretty bad experience overall. Man I wish we used
Amazon SQS...

We have had incidents where it's Master/Slave replication continues on failure
when configured to shutdown the master and slave brokers. It doesn't achieve
the consistency it lists as a feature, and provides zero visibility of
progress. I feel many of the features the project lists are incomplete, and
are listed as a grasp at straws to have the bigger list.

Due to the poor quality of the replication, we had to invest a tonne of time
to implement replication at the block device level with DRBD+GFS2,
complicating the system and (benchmarks pending) likely decreasing
performance, all to achieve a 'feature' ActiveMQ boasts.

I would also challenge a project to have worse, more incomplete documentation.

At the same time, I thank the ActiveMQ team for their contributions and ask
the community to not turn it's back on this potentially decent solution. The
ideas are solid. It isn't production ready, I feel, but that just means it
needs some love.

Cheers,

Tim Vaillancourt

------
anttipoi
I've now been involved in two projects where I would have needed a queue with
following characteristics: \- simple pub-sub, queue-like: m producers, n
consumers \- decent performance (order of hundreds of requests per second) \-
high availability for message persistence (I don't want to loose messages) \-
no strict FIFO needed, just some kind of lesser fairness (newer items should
not block older ones from passing through)

ActiveMQ with active-backup setup over shared disk mount is the current
choice, and the start-up is really slow if the queue has a lot of data.

RabbitMQ does not persist messages in HA fashion, so I've ignored it so far.
Maybe HornetQ needs some attention.

I see a lot of flexibility and feature-richness in the queue landscape and it
perplexes me that getting this simple combination of basics right is so
difficult.

~~~
rbranson
RabbitMQ itself doesn't have HA built-in, but warm spare HA is rather simple
and well-documented, built on-top of Pacemaker and DRBD. It works quite well.

~~~
rabbitmq
Thank-you.

The docs for HA are here: <http://www.rabbitmq.com/pacemaker.html>

This also works with Veritas if you use that instead of DRDB.

Please note that we are currently QAing a new HA model which is active/active.
Watch this space!

------
nvarsj
As a counter point, I have used ActiveMQ for a pub/sub architecture and it
worked great (millions of messages a day, which was benchmarked as only about
10% of potential on a single broker). We did treat it as an API rather than a
deployable component though. We wrapped the broker in our own service
architecture (you can instantiate a broker like any POJO) and disabled
persistency. Queues were used only very lightly. So maybe we dodged a bullet
there... :-).

------
wslh
There is a lot of buzz about open source queue systems. I have tried for
example the RabbitMQ with very bad results. I'll rather explain... one
interesting concept about queues is contention and this is where RabbitMQ and
others are behind. The strange thing is that contention is an old concept from
the old mainframes.

For example the last time I checked you can't block a produced in RabbitMQ
based on the number of messages on the queue.

~~~
rabbitmq
If you have any questions or problems involving RabbitMQ please email us
(info@rabbitmq.com) or post to the mailing list.

"one interesting concept about queues is contention and this is where RabbitMQ
and others are behind"

Can you explain what this means? What kind of contention are you talking
about. You say "RabbitMQ and others" - which others? Who implements this
feature and what does it look like?

"you can't block a produced in RabbitMQ based on the number of messages on the
queue"

Yes you can.

Well - it depends on your use case. RabbitMQ enables you to determine queue
length, and supports flow control.

But, perhaps you had something else in mind?

~~~
wslh
Please let me know where I am wrong:

\- RabbitMQ has a memory based flow control based on the total memory you have
in the machine. I know I can develop something over RabbitMQ to accomplish
what I want (limits based on number of messages) but I prefer to have the
support within RabbitMQ.

\- WebLogic JMS can block producers based on number of messages as a flow
control method.

~~~
rabbitmq
I don't know what WebLogic can do for flow control exactly. Please email
rabbitmq-discuss to get a definitive answer and discussion on what you can do
with Rabbit, which supports various mechanisms.

------
ghostganz
I had a lot of trouble with the network-of-brokers feature in ActiveMQ a few
years ago, which in the end forced us to switch to another solution.

In a later, simpler, project I had issues with ActiveMQ locking up after a
certain number of messages, which was solved with an upgrade.

On a whole, I think the quality of ActiveMQ is lower than other open source
projects with similar brand recognition, e.g. other popular Apache projects.

------
lucraft
One thing I remember in particular from ActiveMQ was that it would often get
itself into the state of having negative queue lengths.

~~~
cmkrnl
The prefetching must be quite aggressive :)

------
PaulHoule
sorta reminds me of the time I built something in 2001 which was a reactive
system that was sort of a web crawler but also a bit more... seems demented
looking back, but I used qmail as a message queue system for it.

~~~
th0ma5
i've considered xmpp as a queue, and hear some do use it!

~~~
rexreed
Yes - I've been seriously considering XMPP for this purpose. I haven't seen
any serious reason not to consider it other than some people's dislike of IM
protocols.

------
rajdavies
BTW - David Pollak didn't talk to any of the ActiveMQ team at FuseSource -
else we would have helped him out - he didn't speak to any of the execs
either. Its a shame he doesn't allow comments on his blog.

------
Maro
I'm not too familiar with the message queue use-case, but perhaps this is
something you could do with ScalienDB, which supports replication and
failover. Disclaimer: I wrote it.

~~~
aaronblohowiak
A message queue is like a database except people can open a socket and say
"notify me when there is a new value for this key" -- and you can chose to let
multiple people get in line for such updates and then wether you want to let
everyone know or just the first person or so-on. you _can_ implement all of
this on any db with polling and locks, but that is not an efficient way to do
things.

------
th0ma5
a lot of what is listed here are all standard caveats of things one needs to
consider in just about any heavy enterprise-java environment.

i think you could sum this up that he simply wasn't working in a situation
that warranted the division-of-labor and configuration flexibility that the
product offers.

------
veyron
How does activemq compare to zeromq?

~~~
andrewcooke
activemq _should_ offer higher level guarantees about reliability. zeromq is
simpler. you can argue that activemq has things you will probably end up
implementing yourself on top of zeromq, or that zeromq has less to get
wrong...

another way of saying the same thing, which illustrates the cultural
differences:

\- activemq is intended to be used in "the enterprise". it tries to implements
a logical ideal, which is a reliable infrastructure that services can use
without being coupled to each other - without worrying about whether messages
were received, or exactly who they go to. to reduce _complexity_ it uses a
central broker (so you send messages to a central "hub").

\- zeromq is intended (imho) for programmers that want to wire things
together. it's less concerned with abstractions and more with providing
something simple clear, simple and flexible that can be understood and used
well. to reduce _latency_ it uses direct connections between peers.

from that viewpoint, you can see that the two are both orthogonal and yet
similar... (disclaimer: i haven't used either, but i used work on an ESB so
have a vague grasp of what's going on. please someone correct me if this is
wrong - i might as well learn as i lose karma ;)

ps rabbitmq is somewhere in the middle and was (i think) originally more
performance-motivated (i believe it's used in finance for example - when speed
might be critical).

~~~
rabbitmq
If you want to know what motivated the creation of RabbitMQ please read
chapter 1 of Jason and Alvaro's book, "RabbitMQ in Action" -
<http://www.manning.com/videla/Videla_MEAPCH01.pdf>

RabbitMQ's main motivation has been to make it easier to join systems
together, scale your applications and manage complex environments. That is
what messaging is for. Back in 2006, we felt there was a need for a good,
stable and scalable open source licensed product that could compete with the
incumbents.

Notice that I did not mention performance. RabbitMQ has good performance and
it is used quite a lot in finance, but the majority of users are what you
might categorise as "anyone using MySQL or Postgres".

Re activemq vs zeromq, I recommend reading "broker vs brokerless" on our blog.

Hope this helps.

~~~
andrewcooke
[http://www.rabbitmq.com/blog/2010/09/22/broker-vs-
brokerless...](http://www.rabbitmq.com/blog/2010/09/22/broker-vs-brokerless/)

(thanks)

------
nerd_in_rage
lol. At a previous job, I used ActiveMQ. It was definitely a flaky piece of
junk.

