
Disque – a distributed message broker - misiti3780
https://github.com/antirez/disque
======
lobster_johnson
This looks great. Are there any plans for topic-based pubsub?

With AMQP (specifically, RabbitMQ) I can set up a fanout topology such that a
publisher simply publishes to an exchange. Clients then bind their queues to
this exchange based on a routing key, and any messages matching the key will
be copied there. If no clients have bound, messages disappear into the aether.

For example, the publisher can use the exchange "changes" to write messages
with keys such as "create.post.123" or "update.user.321". A consumer can then
bind a queue to the exchange with the filters "create. _" and "update._" to
get all create and update messages. A different app can listen to just "
_.user._ " to get anything related to users.

This is a really powerful tool, allowing publishers to be dumb firehoses and
clients to surgically pick what they need. Of course you don't need exchanges
and bindings to accomplish this, only topics.

~~~
seabrookmx
> If no clients have bound, messages disappear into the aether

To everyone who thinks this sounds scary, don't worry. You can bind dedicated
"Undelivered queues" and "Dead letter" queues to exchanges to make sure that
when routing fails, you don't lose any messages.

We're using RabbitMQ in a few newer projects and it's really a joy to work
with!

~~~
lobster_johnson
RabbitMQ is pretty good. Just be aware of its problems with clustering, as
I've outlined here:
[https://news.ycombinator.com/item?id=9448258](https://news.ycombinator.com/item?id=9448258).

------
uptown
Can somebody help me better understand the possible uses for something like
this?

~~~
antirez
Let's say you have a web app that sends emails, and you have many worker
processes actually sending emails. If you use something like Disque, instead
of sending the email directly from the web app, you send a message to Disque,
into the send_email queue: "Please send an email to foo@example.com". Workers
sending emails will fetch from the send_email queue, and will get the message.
Workers sending emails are supposed to acknowledged to Disque when they were
actually able to process the message, so if they fail in some way, the message
is re-queued and delivered again to the same or another worker, after a time
you specify. So basically it adds a reliable middle-layer between the
different moving parts of your application. The user will receive its email,
or during failures it may receive the email multiple times, but it can never
happen that no email is received by the user, since Disque guarantees that the
message will be removed only once acknowledged.

~~~
jonny_eh
Isn't Redis already good for this, powering backgrounding software like Resque
and Sidekiw?

What does Disque offer that's different?

BTW, thanks for Redis. It's one of my favourite pieces of software ever.

Edit: NVM, found this:
[http://antirez.com/news/88](http://antirez.com/news/88)

~~~
revskill
If your Redis died, you must have to saved the sent messages to process later.
It's the most important fact about distributed message queue.

------
djanowski
There's some extra background here:
[http://antirez.com/news/88](http://antirez.com/news/88)

------
sylvinus
FYI, Salvatore will do his first public presentation about Disque at dotScale
on June 8: [http://www.dotscale.io](http://www.dotscale.io)

It will be recorded and posted here as soon as possible ;-)

------
bjblazkowicz
As a redis user this seems very promising due to the fact that I no longer
have to "simulate" message queue functionality.

------
coleifer
I would suggest that one of the reasons so many people use Redis as a message
queue is because they do not want another piece of infrastructure. The fact
that people use Redis as a message queue does not, in my mind, mean that this
is for lack of proven alternatives. I think this is a cool project, but for
me, I will stick with Redis.

~~~
thezilch
I think it's more than that other message queues are "another piece of
infrastructure." In fact, many other message queues are several pieces of
software, and while I can appreciate the modular-nature / separation of
concerns, it can be very overwhelming to introduce, for example, JVM,
Zookeeper, another DB, and the queuing software to handle the above. I'd much
rather have a single, small, and yet still powerful piece of software like
Redis, and if Disque can be that without the performance issues introduced by
mapping a message queue over Redis, that seems to be a huge gain.

This is different enough from Redis that I don't see how they directly
compete. Of course, if you already have a system setup and no performance
constraints with what you have, of course you shouldn't change.

------
djanowski
And there's a Ruby client for it already:
[https://rubygems.org/gems/disque](https://rubygems.org/gems/disque)

~~~
veesahni
Would love to see multi-threaded workers. That's one of the pros of using
sidekiq + redis today (it uses celluloid under the hood).

~~~
heynk
I'd be interested in what it might take to swap out the redis code for disque
in Sidekiq.

~~~
jpgvm
Not fun. Sidekiq is pretty closely married to Redis datastructures
unfortunately.

However there are clones like Shoryuken [1] that would be easier to port.

[1] [https://github.com/phstc/shoryuken](https://github.com/phstc/shoryuken)

~~~
tannerburson
For giggles I spent an hour this evening and added support for Disque to the
pluggable system I help maintain [0].

Things I can say: the ruby disque library isn't really fleshed out yet. It's
alpha, so that's fine, but it's got a ways to go. For example, it doesn't
directly expose ACKJOB as a command. The server is equally alpha, things like
HELP don't do anything yet, and some options on some commands appear broken.
But hey, it was a fun way to spend an hour.

[0][https://github.com/tannerburson/chore-
disque](https://github.com/tannerburson/chore-disque).

------
nasalgoat
Okay, looking at this I can see it as a decent replacement for Beanstalk,
which has been my go-to simple queue server for years now, but has been on
inactive development for some time.

I built a failover system for it involving NFS mounts and heartbeat, but
having it all be automatic would be quite nice. Looking forward to a prod-
ready version.

------
DivineTraube
While this looks promising, why not make Disque part of Redis by introducing
the "message queue" data structure? This would allow to build more powerful,
application-specific messaging abstractions, for instance a compare-and-swap-
key-with-reliable-broadcast (CASKWREB in Redis terminology).

~~~
antirez
It is not possible to replicate Disque features as a Redis data type, it is
fundamentally a different best unfortunately. The project started exactly as a
feature for Redis but was migrated away as a separated system because while I
was working on it the tension between what Redis was and what Disque would be
were bigger every day.

------
ankushio
aphyr is gonna torture this system soon

------
damon_c
Are people pronouncing this word like "Diskqueue"?

~~~
spoiler
I've assumed it's a phonetic play, and that it's pronounced "dis queue" as in
"this queue."

~~~
counterplex
That's close. It's meant to be more like dysfunctional queue.

From the README:

DIStributed QUEue but is also a joke with "dis" as negation (like in disorder)
of the strict concept of queue, since Disque is not able to guarantee the
strict ordering you expect from something called queue. And because of this
tradeof it gains many other interesting things.

------
barrkel
How do people solve the resource allocation problem with distributed job
queues?

By resource allocation problem, I mean that jobs may be small (so that lots of
them can occur in parallel) or large (occupying a significant fraction of a
machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying
to do too much can effectively crash a system with OOM killer or paging.

Does everybody just roll their own resource-based scheduler?

~~~
tobz
In the case of RabbitMQ, it starts paging messages to disk, IIRC. If you run
into descriptor / memory / disk limits, it starts turning away new messages
until it returns back to a safe threshold.

There's a certain amount it can do, but ultimately, you have to apply
backpressure at some point unless you're willing to start losing messages at
the message broker layer.

After that, there's still a lot you could do: spool them to disk on the
publisher, retry later, etc. You still risk message loss, filling up THOSE
disks, etc... but again, something has to eventually give.

~~~
barrkel
I'm not worried about too many messages piling up; I'm worried about the
consumers pulling too many messages off and acting on them concurrently using
up all the resources on the machine, or pulling off too few messages, and
leaving too many resources idle.

------
mmgutz
How is this different than beanstalkd?

~~~
nasalgoat
Good question - I've been using Beanstalk to handle hundreds of thousands of
worker transactions per second for years.

I suppose development on it has slowed to a crawl so it's no longer "actively"
supported.

------
bryanlarsen
previous discussion on the blog post announcing Disque:
[https://news.ycombinator.com/item?id=9208501](https://news.ycombinator.com/item?id=9208501)

------
JulianMorrison
I think the thing this is most comparable to is NSQ, but NSQ takes a different
approach to distribution.

\- disque: send to one, read from one. The message is handed off to N
replicas, and efforts are made to avoid duplicated or dropped messages. Disque
will refuse new messages if RAM is filled across the cluster.

\- nsq: send to any. Read from all. NSQ nodes do not communicate or coordinate
with each other. Since only one node originates the message, there's no
duplication, but a node outage can drop messages, and a partition can isolate
them with no consumer. NSQ can grow its queue beyond RAM, so it will keep
accepting new messages even if it is partitioned from the consumer.

Personally I think NSQ's approach looks like it's doing a lot less work and
achieving almost all the same guarantees.

------
fweespeech
Tbh, I'd have preferred a synchronous replication option for Redis nodes. That
seems to be essentially the "improvement" in Disque and it'd be easier to
maintain two replication nodes than two separate projects imo.

~~~
antirez
Hello, Redis + sync repl is a valid point of view IMHO, there are reasons why
this was discarded, but it is definitely an option. However Redis + sync
replication does not give you Disque at all. Disque is a specialized
distributed system hard to simulate with Redis, nodes runs protocols in order
to coordinate re-queueing the message, in order to federate to route messages,
and so forth.

~~~
fweespeech
Fair enough. :)

I am basically using something I could fit in Redis + sync replication [in
terms of data model / function] as a job queue presently so I suppose that is
just where my mind jumps to.

------
urlgrey
I've created a Go client for Disque, available here:
[https://github.com/zencoder/disque-go](https://github.com/zencoder/disque-go)

------
bhz
Looking forward to giving this a try - we currently use an offshoot of
"resque" for the Java ecosystem called "jesque", but it's always good to
explore alternatives.

------
onyxraven
It seems like a logical extension/simplification of Redis for a common
usecase, as he said.

I'll be interested in seeing comparisons vs other mq systems.

------
mmastrac
I'm excited to see what comes out of this. I'm a huge fan of Redis, mainly
because it gives you a bunch of really tiny, easy-to-understand primitives
that you can combine to make something very powerful.

I don't know if this will follow the same philosophy, but I'll be keeping a
close eye on this as it evolves.

~~~
djanowski
I would say that Disque follows the minimalistic philosophy of Redis. However,
Disque is a specialization of one of the most common use cases of Redis:
queues. So I wouldn't expect so many primitives--Disque knows about jobs and
queues, so you don't have to build those yourself like people have been doing
on top of Redis.

------
jvehent
Any functional advantage to using this instead of something like rabbitmq?

~~~
lobster_johnson
As a RabbitMQ user, I'll switch to the first viable alternative that is
production ready.

RabbitMQ's clustering isn't great. It's sensitive to partitions, which can
occur not just from actual network hiccups but also simply due to high CPU or
I/O load, and it does not have a good strategy to recover from such
partitions.

RabbitMQ is not multi-master by default. A queue is owned by a specific node,
and if you have a partitioned cluster, that queue (and related objects such as
exchanges and bindings) will disappear from other nodes.

You can patch RabbitMQ's clustering by enabling "high availability", which is
their term for mirrored queues. Each queue will get a designated master, and
be replicated to other nodes automatically. If a partition happens, the nodes
elect a node to become a new master for a mirrored queue.

Unfortunately, this opens the cluster up to conflicts. Let's say you get brief
partition. Now all the nodes see each other again, and you have conflicting
queues: Node A used to be master of queue X, now node B is also master of
queue X. During the split, their contents diverged a little bit. But RabbitMQ
has no way to consolidate the two masters, so the queue is not operational.

To fix this, either you need to reconstruct the queue manually (usually
impossible from an application's point of view), or wipe it (hardly a
solution) or simply have RabbitMQ automatically pick a winning master and
discard the other master(s). This mode is called "autoheal", and picks based
on which master has the most messages; the previous master(s) are wiped and
become slaves. This is coincidentally the _only_ mode in which RabbitMQ can
continue to run after a partition without manual intervention.

In practice, recovery has proved flaky for us. Nodes stay partitioned even
after they should be able to see each other. We have also encountered a lot of
bugs — for example, bindings or exchanges disappear on some nodes but not on
others, or queues are inexplicably lost, or nodes otherwise just misbehave.
We're on a cloud provider which is otherwise rock solid; of all the software
(databases etc.) we employ in our clusters, RabbitMQ is the only one that
misbehaves.

This is anecdotal, of course. Fortunately, the author of Jepsen, Kyle
Kingsbury/"Aphyr", has done the maths to back this up, demonstrating that
RabbitMQ's clustering is both theoretically and practically unsound [1].

This may be overly harsh. RabbitMQ is a decent project. RabbitMQ has a lot of
features. Things like routing keys, flexible durability, TTLs and dead letter
exchanges are great [†]. When it works, it works really well. But in the real
world, I wouldn't want to run it more than two nodes, and preferably not at
all.

[1] [https://aphyr.com/posts/315-call-me-maybe-
rabbitmq](https://aphyr.com/posts/315-call-me-maybe-rabbitmq)

[†] Although unfortunately DLXes are effectively unusable in a looping
topology configuration (ie., for timed retries), as AMQP frames will increase
indefinitely in size.

~~~
tobz
How do the AMQP frames increase in size indefinitely? (I use DLXs and message
TTLs to do timed retries, without issue, although at a very low retry rate, so
maybe I'm just not hitting your purported issue with increasing frame sizes?)

~~~
lobster_johnson
Every time a dead-lettering happens, RabbitMQ will do this [1]:

 _The dead-lettering process adds an array to the header of each dead-lettered
message named x-death. This array contains an entry for each time the message
was dead-lettered._

This table is never truncated. It exists in the AMQP frame and will grow
indefinitely.

The AMQP spec mandates that clients negotiate a frame-max size per connection
[2], and I believe RabbitMQ is in violation of the spec through this behaviour
if the client specifies a non-zero value. (RabbitMQ even ignored this
negotiation prior to 3.1, and there's a document [3] which indicates it
ignores limits entirely.)

As a result, clients that follow frame-max strictly according to spec (such as
amqplib for Node.js [4]) will refuse to handle violating frames, which is
understandable given that clients also like to have predictable buffer
allocation.

[1] [https://www.rabbitmq.com/dlx.html](https://www.rabbitmq.com/dlx.html)

[2]
[https://www.rabbitmq.com/specification.html](https://www.rabbitmq.com/specification.html)

[3]
[https://www.rabbitmq.com/amqp-0-9-1-errata.html](https://www.rabbitmq.com/amqp-0-9-1-errata.html)

[4]
[https://github.com/squaremo/amqp.node](https://github.com/squaremo/amqp.node)

------
marcosnils
First version of the Java client available here:
[https://github.com/xetorthio/jedisque](https://github.com/xetorthio/jedisque)

------
ybrs
if anyone wants to play with, i wrote a client for python,
[https://github.com/ybrs/pydisque](https://github.com/ybrs/pydisque)

------
djanowski
And there's a Ruby client to start playing with it:
[https://rubygems.org/gems/disque](https://rubygems.org/gems/disque)

------
falcolas
Interesting. Gearman with a few additional promises (sudh as providing
transactional and durability guarantees). I'm looking forward to seeing this
mature.

------
agopaul
Can wait for the PHP client so I can finally drop Gearman

~~~
orthecreedence
Out of curiosity, have you tried beanstalkd? What did you not like about it?
Disque is very similar to beanstalkd, but with distribution built in.

~~~
agopaul
I haven't tried it actually. I evaluated it a few years ago and I remember
that the queue persistance was missing (though now I see it is supported), and
it lacked handling retries out of the box (which Gearman does).

Now I've wrote my own wrapper to handle retries on the client side so I might
give it a try.

Edit: Grammar

~~~
pentium10
huh that was some years back, it has automatic retry and mature protocol.

------
cdelsolar
This is SUPER awesome. I can't wait till it's production ready but maybe I'll
play around with it a bit on side projects.

------
jlangenauer
Importantly (not really), is it to be pronounced "dis-queue", or "Disk"?

------
JSno
Is this one trying to do something that Kafka does? Or this is a different
beast?

------
lost_my_pwd
Slightly earlier post here:
[https://news.ycombinator.com/item?id=9447185](https://news.ycombinator.com/item?id=9447185)

~~~
Alexandervn
Link goes to this page.

~~~
corobo
Two posts were merged, that comment came in from the other one

------
itamarhaber
Congrats @antirez!

