

Introducing Resque - mrduncan
http://github.com/blog/542-introducing-resque

======
xal
I wrote DJ. This looks amazing for high volume sites. At Shopify DJ is barely
able to keep up these days.

My biggest regret with DJ is actually that I added numerical priorities
instead of choosing the same tag based approach that Resque implemented. I'll
likely release a DJ2 at some point which backports some of the ideas.

~~~
trevorturk
FWIW the author of DJ posted in more detail on this subject in this Google
Group:

[http://groups.google.com/group/delayed_job/msg/9e2987caa177b...](http://groups.google.com/group/delayed_job/msg/9e2987caa177ba2d)

~~~
mattyb
FWIW, xal (who you replied to) == the author of DJ.

<http://github.com/tobi/delayed_job>

------
tptacek
The Resque code aside --- and I want to switch to Resque, like, tomorrow ---
this is the best article on Ruby job queues I've ever read. It's Github's
review of pretty much every mainstream job queue out there. Excellent.

------
bjclark
Redis is almost perfect for a queue because of it's lists and atomic
operations. And, most of all, because it's persistent. It's amazing how many
queues aren't persistent.

I'm glad someone finally did this, it could be a great solution. I'm looking
forward to digging into this.

~~~
mrduncan
_And, most of all, because it's persistent._

Just a slight caution, since Redis stores it's data in memory and
asynchronously writes out to disk it is possible to lose small amounts of data
in a crash. In most cases this is an acceptable trade off for the speed of an
in-memory database though.

There is also plenty of good info on the project page -
<http://code.google.com/p/redis/>

~~~
antirez
Since Redis 1.1 (currently in beta - don't use it in production for now) there
is a new append-only journal storage system alternative to the main one (that
is, snapshotting), so starting from 1.1 Resque will be more reliable as well
if needed, because in this new (optional) configuration Redis can't lose data.
It's also possible to configure if it should fsync() never, everysec, or after
every write. It's up to you to select the tradeoff between speed and safety :)

~~~
jeremyw
Fantastic. Redis can go in both my ephemeral and critical systems.

You're a whirlwind of activity.

------
jbyers
Nice. We built a mysql-backed queue with nearly identical properties but it's
not especially efficient. It does 10+ jobs per second per worker, but with
locking this starts to cause InnoDB concurrency problems. Not insurmountable,
but moving to a Redis-backed approach seems like a great option when our
current queue can no longer keep up. Really glad to have this approach proven
by GitHub.

------
dacort
Really looking forward to trying this - I've given backgroundrb, BJ, DJ, and
workling a go and there's always been something that's frustrated me.

It's amazing the number of different things I'm considering using Redis for as
well.

------
adriand
I've also been through a lot of different background processing libraries for
Rails, and so far, if you have simple needs, I'm finding that Spawn is really
awesome. <http://github.com/tra/spawn/>

I think something like Resque will be very helpful to us over the long term,
but if you're looking for something quick that just works to send a big batch
of emails, import stuff via a web service, whatever, check out Spawn. Not at
all appropriate for a site like Github, but works well for our vastly smaller
websites.

------
richcollins
Also see VertexDB (<http://github.com/stevedekorte/vertexdb>).

It is a high performance persistent graph database with filesystem like
semantics (will soon support FUSED). It presents an API for an atomic queue
pop operation, which allows for synchronization of jobs.

------
paraschopra
Slightly off-topic, but is there any stable background job queue available for
PHP (and perhaps MySQL)? I have been using custom hacked solution on top of
tokyo tyrant but I will prefer if someone has a mature library.

------
callmeed
Jeeze, I just started migrating from BJ to DJ ... what should I do now?

