

An OSS scalable distributed message queue with automatic failover. It kicks ass - dhruvbird
http://code.google.com/p/pymq/

======
plq
this recent mq hype is really about the unification of a directory server, a
wire protocol and a reliable (or not) message transfer scheme just like the
good old smtp. once one sets up a proper directory solution (ldap) and uses
thrift for serialization, the need for a messaging daemon goes away -- every
publisher can keep an (optionally) persistent queue however they want.. so the
mq becomes just a library.

it seems to me that imatix guys have been there, done that and saw that mq-as-
library was the way to go for a more efficient and reliable distributed
transaction processing system. <http://www.zeromq.org/whitepapers:brokerless>
is an interesting read in this regard.

~~~
shykes
I am going to read this avidly. ZeroMQ's broker-less approach seemed like more
work for me. This is an opportunity to reconsider my position.

We use an AMQP stack and the complexity seems somewhat unjustified. On the
other hand, it's good to have a standard, even if it's not perfect.
Django+celery is pretty kick-ass, and I would have to give it up if I moved
away from AMQP.

~~~
dhruvbird
Celery is pretty cool in it's own way. It is like the appengine Task Queues
rather than message queues in general.

------
hawthornenigel
From the clustering page of the Wiki:

"The pymq cluster is not designed to survive a master crash. If the master is
to crash, all bets are off. However, clients that are already connected to the
individual nodes can continue to operate normally. It's just that automatic
failover and queue discovery won't work (which is basically what the master
does)."

This is not fail-safe, unlike most enterprise messaging systems. Though you
could stick the master on a VMware cluster with FT enabled and survive
hardware failures, there is nothing preventing data loss from software or
networking failures.

~~~
dhruvbird
The assumption is that the master isn't flaky. You can achieve this in
multiple ways. One of them you mention.

If the software is bad, you can't help it.

As far as network failures are concerned, I think those are handled. If your
network fails, you basically won't be able to send a message or won't get
confirmation of it being sent. Either ways, if the queuing system works in at
least once mode, you are guaranteed message delivery. If there is an error,
the library reports an error and the application must re-try.

------
sandGorgon
how does this compare with Beanstalk ?

~~~
dhruvbird
Beanstalkd is a workqueue service not a message queue service. It's use case
is some what more in tune with CRON rather than pymq.

