
Disque 1.0 RC1 is out - antirez
http://antirez.com/news/100
======
kevan
> However I’m not living into the illusion that I got everything right in the
> first release, so it will take months (or years?) of iteration to _really_
> reach the operational simplicity I’m targeting.

It's always refreshing to hear such good programmers acknowledging how hard
building complex systems is.

~~~
BogusIKnow
And this from antirez who writes such excellent programms, Redis runs in
several companies I've been, without ever crashing, making any trouble - most
people I've met forget they have Redis running because it just works and works
and works. Can't praise that piece of code high enough.

~~~
Mahn
Indeed, we have had redis running in production for four years now and I can't
remember a single crash happening. If something goes wrong in our stack redis
is the last thing we check. Antirez should write a http web server :-)

~~~
BogusIKnow
+100 'Antirez should write a http web server'

------
lobster_johnson
Disque is definitely exciting, and looks like it can replace RabbitMQ, which
has serious flaws in its clustering design. I'm looking forward to trying it
out.

However, if some constructive criticism is permitted, I have to say that,
having written distributed applications for many years, I have come to dislike
the "classical" push/pop queue data model:

* Acking is a bad idea. It requires the broker to manage a lot of state, including locking and timeouts.

* Re-queuing invalidates total ordering.

* On the performance side, parallel distributed queue consumption (which also breaks total ordering) is directly at odds with this model.

* Queues as opaque objects — you can only inspect by popping the top message, and you cannot access older, dequeued messages. Fortunately, Disque allows you to read the entire queue without mutating it, but it doesn't look like you can read old messages.

* Complicated queue topologies (fanouts, dead letter queues, etc.) become a logical necessity of the strict FIFO structure. (These topologies need to be declared every time the client starts up, and introduces the possibility of schema conflicts.)

* Logical de-duping is probably not possible.

Apache Kafka gets the data model right. It wisely acknowledges that queues are
linear and should stay that way: In Kafka, queues are strictly append-only
logs where every consumer has a cursor to the last position it read. In this
model, many of the classical concerns melt away: Acks/nacks are unnecessary
(consumers simply "commit" their position); total ordering is always preserved
(since the queue cannot be reordered) and parallelism is made explicit
(through named partitions); de-duping is trivial, and complicated topologies
are largely eliminated (AMQP-type "exchanges" that fan out to separate queues
are unnecessary because multiple readers can all consume the same queue
without changing it, as their position is independent of the queue); and you
get to choose either at-most-once or at-least-once delivery consistency by how
carefully you manage your offset.

Since logs are strictly linear, you are also given the choice of how much
history to keep — all of it, if you want — which opens up some interesting use
cases that are not possible with classical brokers.

Kafka isn't perfect. It's a huge pain if you're not in Java land. There are no
modern, mature "high-level" client implementations for Go, Ruby or Node.js.
Its reliance on the JVM and on ZooKeeper makes it fairly heavyweight, both on
the server and on the consumer side (the new API for broker-stored offsets
simplifies things, but non-Java clients are far behind). I would really love
to have a lighter-weight, language-agnostic Kafka-like implementation without
the Java baggage.

Last point: The fact that Disque calls its messages "jobs" makes me a little
disappointed that it is, in fact, not a job management system. I'd love a
solid, distributed job manager.

~~~
Jasper_
Kafka has a different data model which works for some scenarios, but not
others.

We attempted to use Kafka as part of a job management system, where long-
running jobs were scheduled and workers consumed partitions, but what we found
was that since consumers work on partitions, a long-running task could block
an entire partition's worth of work, with no way to migrate it to another
partition.

Kafka works really well when the bottleneck is the broker to begin with -- if
you have a lot of small, lightweight messages being passed to other systems,
and the broker is having trouble keeping up. Analytics and logs are great
examples where Kafka's data model works really well.

But for a smaller number of larger messages that takes seconds to minutes to
process (we were using it to schedule data downloads and video processing),
workers could be sitting idle while there was still plenty more work to do.

We moved to Amazon SQS for now, but I imagine we'll move to something like
Disque in the future.

~~~
lobster_johnson
I would argue that job management is unrelated to messaging, at least
according my loose definition of a task that:

* Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed;

* Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior;

* Has state data (e.g., completion % progress, log output, metrics, transactional continutation state for pausing/retrying);

* Can be scheduled on multiple nodes;

* Can be created and scheduled multiple times;

* Can be scheduled by priority;

* Can have its resources (RAM, CPU, I/O) constrained.

The thing is, queues are terrible at this. People continue to abuse messaging
systems as a scheduler, and it always sucks.

For example, consider acking: For a long-running tasks that might take hours,
do you really want to hold up the broker's internal lock for all that time?
What if you restart the broker — the task is still running, now you decoupled
its running state from the queue. Similarly, queues are terrible at finding
out what's waiting, or what has happened. By using the queue as job _state_ ,
you are coupling scheduling with execution, which is just wrong.

A queue (one that supports priorities, mind you) is, however, excellent for
management _commands_ : "Start job X", "pause job X" and so on. But you will
want to maintain execution state separately from the queue. A relational
database is quite good at this, though one could use a NoSQL databases, too.
First, create a row representing the job: {id="job1", state="new",
type="import_stuff", params="..."}. Then push a queue message
{commmand="start_job", job_id="job1"}. Run a queue consumer or ten on each
node. Each consumer acks the message, finds the job in the table and starts
it. One could run it as a child process, or in a Docker container. One could
of course use an orchestration service (like Mesos/Marathon) to run it, and
monitor the service for container events and update the job table
appropriately.

I would want to take it further and allow the job to update itself through
API, so that it could publish metrics and log messages, as well as maintain
whatever data is useful snapshotting. For example, a long-running task to
import data into a database would benefit from maintaining a cursor so that
you could pause/kill the job and later resume it at the point where it was.

~~~
Jasper_
Sure, that's actually very similar to the the system we built internally. But
I don't see how Kafka as a messaging layer helps in your system, either.
Presumably, the consumers won't advance their offset in their partition until
the job is done, because they are not ready to start a new job. So you will
still get the same blocking behavior.

In addition, I don't see how you implement priority in a system like this,
since Kafka partitions are append-only. One has to process the former
start_job messages before you reach the higher priority items.

Having a broker hand out items in priority to workers makes a lot more sense
to me.

The job system you described is definitely something missing from the OSS
world, and we'd love to open up our system for this eventually. While Disque
is not that piece, I think it makes more sense than Kafka.

~~~
lobster_johnson
I didn't mean to imply that Kafka was suitable for such a system, though
that's mostly because it doesn't support priority queues. You could work
around this by using a fixed range of levels and one queue per priority level,
though it would probably just complicate things.

------
bipin_nag
Looking forward to Aphyr's Call Me Maybe for Disque.

~~~
deanclatworthy
This was my first thought also. aphyr seems to be the go to for finding flaws
in _everyone 's_ distributed systems.

~~~
jbaiter
If you look at the last paragraph, it seems aphyr has been involved in the
development:

> I was not alone during the past months, while hacking with Disque and trying
> to figure out how to shape it, I received the help of: [...] _Kyle
> Kingsbury_ , [...] Redis Labs and Pivotal, and probably more people I’m not
> remembering right now. Thank you for your help.

------
ksec
Idle observation, The blog shows what simple, text, and minimal graphics /
pics ( In this case there are Zero ) can load incredibly fast...

------
yowmamasita
Q: Can I use this on top of any non-ACID-compliant database to avoid race
conditions?

~~~
joneholland
No.

~~~
vhost-
Why?

~~~
ykl
See the CAP theorem [0]. Disque is designed to guarantee availability and
partition tolerance, which according to CAP means Disque cannot guarantee
atomic consistency (the C in CAP is not quite the same as either the A and C
in ACID [1]) between nodes. No atomic consistency means no guarantee of a
single action to single response correspondence, which means race conditions
are possible.

[0]
[https://en.wikipedia.org/wiki/CAP_theorem](https://en.wikipedia.org/wiki/CAP_theorem)

[1]
[https://en.wikipedia.org/wiki/Consistency_(database_systems)](https://en.wikipedia.org/wiki/Consistency_\(database_systems\))

------
willcodeforfoo
Neat, love reading antirez's posts and the README for this is amazing.

------
navaneethkn
This looks interesting!

Currently, we have a huge background processing system built using Sidekiq
backed by Redis. The biggest problem we are facing today is to run machines
with lot of in-memory storage because jobs are queued to Redis and Redis works
in-memory. These machines costs a lot.

I think the same issue will happen even here because this is also in-memory
store and persistence works like how Redis handles it.

------
Dowwie
This is exciting. Disque is intended to replace redis in celery (python). I'm
looking forward to using it!

------
leeoniya
good question in the blog's comments:

How is this compared to 0mq or nanomsg?

~~~
lobster_johnson
ZeroMQ is not a messaging broker, it's a socket abstraction. (I interpret the
zero in ZeroMQ to mean "no".) So they are not comparable at all. You can build
a broker with ZeroMQ, which people have done [1], in the same sense that you
can build a broker on plain TCP sockets.

I believe the same applies to Nanomsg.

[1] [https://github.com/zeromq/malamute](https://github.com/zeromq/malamute),
[https://github.com/sintaxi/waterfront](https://github.com/sintaxi/waterfront),
[https://github.com/zeromq/zbroker](https://github.com/zeromq/zbroker)

------
vdfs
Is there a product-ready alternative to Disque?

~~~
AznHisoka
I've been using Sidekiq for the past 2 years, and deem it as production ready.
It does force you to use Redis and Ruby though.

~~~
tbrock
I'm building a node.js application now and __WISH __we had something half as
good as Sidekiq written in Javascript.

~~~
true_religion
How would you say it compares to something like Celery?

~~~
tbrock
I found sidekiq to be better than celery for my needs because it was better
integrated (although less configurable). It did what it said on the box
without a fuss and without me having to debate the merits of and choose a
backend, etc...

It worked with redis and it worked well.

