

Introducing Disque [video] - sylvinus
http://www.thedotpost.com/2015/06/salvatore-sanfilippo-introducing-disque

======
antirez
$ ./antirez --voice "godfather" \--talk

(I hope is understandable at least...)

~~~
linkmotif
So nice to see you post this here. While watching I was really enjoying your
Italian accent and was thinking about how amazing it is that so many people
have to learn English. Reminded me of your post "English has been my pain for
15 years" ([http://antirez.com/news/61](http://antirez.com/news/61)). Thank
you for conquering the language and presenting this to us. Really glad to have
another message queue to explore.

~~~
antirez
Thanks linkmotif, when I finished talking, as usually, there were people
literally ROTFLing for the accent, but in a positive way. Finally at least I
can kinda talk in a more casual way, it used to be harder than this in the
past because of totally lack of practice, usually I just write/read. Thanks
for your message :-)

~~~
ie
I am a native english speaker (born/raised in California) and also very self
conscious about the way I sound for different reasons. I suspect many of us
have this kind of internal battle, even if it seems relatively insignificant
to other people.

FWIW, I think your accent is interesting and enjoyable to hear.

~~~
antirez
Thanks :-) I understand what you mean I think, since speaking Italian I've
also a strong Sicilian accent... but I tried hard to preserve it, after all
it's my origin.

~~~
mandeepj
+1 for not forgetting your roots. I am also in a battle with my accent.
Hopefully, one day, very soon, I will win it.

------
tempVariable
Thank you for providing yet another project to the open source community
Salvatore.

~~~
antirez
Thanks! My pleasure, and I'm already receiving tons of help from the
community.

------
matthewmacleod
I'm pretty excited – I've spent a couple of weeks on and off trying to build a
useful reliable job queue system on top of Redis, but it's not really working
as simply as I'd like.

Disque seems to offer the features I am interested in along with the
relatively simple semantics of Redis. Looking forward to the API being settled
a bit so I can play with it.

------
yodon
For anyone curious, Disque is a synchronously replicated in-memory message
broker/job queue that was designed to capture the essence of the "Redis as a
jobs queue" use case.

------
arturhoo
Hi Antirez, always nice to use and see work from a good professional and
outstanding community member. Whenever I saw people using Redis as a queue
server, it was mainly for two reasons: 1) they were already using Redis and it
had the right primitives, 2) it was dead simple to setup and use (compared to
rabbitmq, for example).

One guy that I think is often overlooked is beanstalkd
([https://github.com/kr/beanstalkd](https://github.com/kr/beanstalkd)),
especially with regards to 2). Have you ever taken a look at the project and
if so, how do you think it compares to your vision for Disque?

~~~
antirez
As the author of one of the two systems I don't feel super well at comparing
Disque and Benstalkd, but I guess, as Disque matures, people informed about
both systems will compare them. However an obvious difference is that Disque
is "internally" distributed and federated, while Benstalkd requires the client
to take care of managing multiple nodes.

------
yazaddaruvala
antirez as you mentioned both projects (Redis, Disque) are based on the same
"framework" (skeleton).

Do you think because of their different guarantees that shared code will
diverge?

Do you think that base layer will have to be kept in sync? (bug fixes,
improvements)

I guess I'm asking, if at all, at what point would you pull that skeleton out
into a separate repo and share it across Redis and Disque?

~~~
antirez
Initially I had to face this choice, and I decided to go for the independent
fork for a couple of reasons:

1) As you said, my guess was that the core part would diverge dramatically, if
not for the separated files that can be kept in sync anyway, like event loop,
string library, and so forth. And the time is showing this assumption was
true. The code is diverging and as it diverges the effort to take it under the
same library may become complex.

2) This was even important than "1": the effort to rewrite Redis in terms of a
common core was a big one. I was not able to stop the development to undertake
this effort, just for a side project which was not clear at the time where was
headed. Now I'm more confident that Disque will get my attentions for the next
years, but still I don't have the time to attempt such a big refactoring /
rewrite.

However I cherry-pick from the common parts, and do some "diff" from time to
time to see if there are ideas in one side that can be merged in the other
side. Is not optimal, but I had no choices. What can be done in the future is
to turn more parts of both systems into common libraries that can live in
separated files, and rewrite only those parts, incrementally.

------
shockzzz
I like Salvatore.

~~~
sehr
I'm almost ashamed I didn't know his real name before, but I guess his
projects speak for themselves!

------
bluecmd
I find it interesting that the article claims eventual consistency, but also
best effort ordering. The consistency in a queue is surely made of two parts:
count and order. To not have duplicates is nice, but if ordering is not
guaranteed - is this really a queue? Seems like it's a Something In Something
Out, or "a pile" or something. Again, curious about what the evental
consistency guarantee is referring to.

Not saying it cannot be useful - for work distribution it will surely work -
but it needs to be considered for your usecase that you will need to be able
to process messages in an unlimited unordering. Process a delete action before
a create for example.

~~~
antirez
What is eventually consistent is the job state: it means that jobs are removed
only when they are successfully delivered at least one time.

Best effort ordering does not mean no ordering btw. Use cases for which strict
ordering is needed are rare, and it is possible to build order out of an
unordered queue if strictly needed (especially since, when strict ordering is
needed, usually you have a _source of truth_ to maintain the state of the
application, that happens to be a strictly consistent store).

The cost of ordering is huge in terms of design, and benefits just an handful
of use cases. So I think best effort ordering is one of the best Disque
assets.

~~~
bluecmd
Ah OK. This might sound harsh, but so the guarantee you have is that the data
will not be lost? That's a pretty weak guarantee no?

Also, best effort implies no ordering in the worst case no? I always try to
judge systems when the world isn't peaches.

------
mattkrea
As someone who has to maintain a RabbitMQ cluster.. I cannot wait to be able
to use this reliabily.

------
alphadevx
How does this compare to Resque (I've used that in production and like it a
lot)?

~~~
antirez
Resque is based on Redis, so it is similar but can't offer the same
guarantees, scalability and fault tolerance. However for many use cases where
ephemeral messaging is enough, Resque is a good pick.

~~~
alphadevx
I love Redis and I love messaging systems, so will definitely take a hard look
at Disque, thanks for all your hard work!

------
nodesocket
Does Disque support recurring jobs, like queue this task, and run it once
every 4 hours. I know Antirez mentioned there is a delay property in the
video.

Also will there be support for absolute scheduling like, run once at 4:30:00am
today.

~~~
rakoo
What more would it have than a simple script in a crontab that pushes the job
every X ?

------
guiomie
Looks interesting. But why would I use this instead of RabbitMQ ? (I didn't
watch the video, but I did check the github repo)

~~~
matthewmacleod
I reckon it'll be a complexity thing.

Disque uses the same kind of tremendously simple API as Redis does. It appears
to have ~15 commands, and is basically just a job broker and nothing else. If
it gains traction, it will solve a specific common problem well.

RabbitMQ is a more comprehensive message broker, and more complex. It ships
with federation, an HTTP API, SASL, that sort of thing. Useful tool, but
likely more complex to setup and administer (to the point that it might be
excessive for some use cases).

Obviously this is all subject to personal suitability though!

