

Resque Bus - bleonard
http://tech.taskrabbit.com/blog/2013/09/28/resque-bus/

======
est
One thing I really hope with Redis's PUB/SUB is that it could support a time
parameter, so we could develop a cronjob replacement, could be add/delete
dynamically, and broadcast, and managable with API or Web UI.

Currently all redis-based background job implementations does not support
scheduled jobs.

Redis is not only a memcache replacement, but also a good data structure
server, as central hub I hope it add more time-sturcture functions.

~~~
aaronblohowiak
>Currently all redis-based background job implementations does not support
scheduled jobs.

That has been false for 4 years.

[https://github.com/resque/resque-scheduler](https://github.com/resque/resque-
scheduler)

~~~
est
It's polling based approach.

[https://github.com/resque/resque-
scheduler/blob/master/lib/r...](https://github.com/resque/resque-
scheduler/blob/master/lib/resque/scheduler.rb#L36)

[https://github.com/resque/resque-
scheduler/blob/master/lib/r...](https://github.com/resque/resque-
scheduler/blob/master/lib/resque_scheduler.rb#L225)

Resque will scan for set value with key prefix like "timestamps:" or
"delayed:", for every five seconds, if there's any task queued it will be
fired.

There's many pros/cons with this approach, e.g. in Redis SADD, SMEMBERS are
O(N) operations while in theory dedicated cron algorithm could make O(log(n)).

[https://en.wikipedia.org/wiki/Cron#Multi-
user_capability](https://en.wikipedia.org/wiki/Cron#Multi-user_capability)

This is is important if you have many short lived, timestamp based events need
to handle. My use case is design item cool down for a mobile/web game using
server side push, with millions of users. While resque could in theory solve
this problem, but it's no different from a per 5 minute crontab command.

So anyway, what I hope is a native clock in Redis handling all this.

------
agopaul
How do you handle exceptions and errors who would require a rework of the job?

That is interesting, I was looking for something similar to an EBS for PHP
time ago, but eventually I wrote myself an ad hoc solution which uses MySQL
(and row locks) plus a gearman worker and cron.

~~~
bleonard
Resque Bus uses resque-retry and resque-scheduler to do exponential backoff.
That is, it will retry immediately, 2 seconds later, 4 seconds later, and so
on up to several hours. It was a little tricky to make sure it ended back in
the right application queue, but it worked. If it was put in another one, it
would crash because the code to process it would not line up.

------
bradhe
This is truly awesome. I have been wanting this semanticto supplement my
queues for a long time and was dressing adopting an operationally more
complicated tool like Kafka, not to mention all the effort that would be
needed to port code.

------
sjtgraham
Why not use AMQP?

~~~
bleonard
Edited a comment from the blog comments a bit...

I tried my best to explain some of the reasons we went this way instead of
using things out there. I'm not claiming to have invented anything
particularly novel here because it's a simple problem to move things from one
list to N lists based on keys. We found value in doing it with technology that
was already part of our stack and very well known amongst the team and
community.

I did an implementation in RabbitMQ at the outset. I found that the simplicity
of just moving the data around, clarity of persistence, and control over
subscriptions was worth the relatively short time it took to implement this
gem. For example, we have found value in matching on any of the or multiple
published keys (specific values, regex, blank, present, not present, etc) as
part of the subscription model. I could not figure this out using the
routing_key and/or topic kind of stuff in RabbitMQ.

------
hcm
Check out Hutch
([https://github.com/gocardless/hutch](https://github.com/gocardless/hutch)) -
a similar system built on top of RabbitMQ

~~~
bleonard
That is close. Thanks for the link.

The Consumer mixin is almost the same as the Subscriber one that I came up
with.

Is there a way to subscribe on more than just the topic type? For example, to
anything published that has { action: 'received' }

That's something that really enables some great use cases for us.

------
agerlic
Good example of resque/redis ecosystem usage. It could be great to have more
info about your failover and monitoring strategies.

