
Rescuing Resque again (2015) - polysaturate
http://words.steveklabnik.com/rescuing-resque-again?hn
======
thibaut_barrere
In my opinion, Resque so far failed because like a lot of OSS projects (and
unlike Sidekiq), it isn't built on sustainable economical grounds - from the
maintainer POV.

Bringing outside "funding" can work for a while (see JRuby), but when the
funding stops, it only underlines that the financial engine should really be
carefully "designed" at the core of the OSS project, for things to be
maintainable on the very long run. Otherwise history is bound to repeat
itself.

Large projects can enjoy a large pool of contributors, sometimes with less
risks of burn-out (often seen on small OSS projects around me at least), each
contributing marginally, with a few paid core members (either directly or
indirectly via by-products of a well paid activity), but for smaller projects
like Resque, it's more likely a single individual full-time, paid by the
project.

Again Sidekiq is really a great inspiration here for me (disclaimer: I plan to
launch a "pro" offering for an OSS project I manage at [http://kiba-
etl.org/](http://kiba-etl.org/), and maintained another open-source for a
while before which ultimately went to the cesspit).

~~~
steveklabnik
Yup, this is exactly it. The only reason I was able to do all the open source
work I did was because of my job at Jumpstart Labs, and once that ended,
well...

That said, as you mentioned, this is pretty much the default state for the
large, large part of open source projects. Something's got to give somewhere,
eventually.

~~~
thibaut_barrere
What is your take on the Sidekiq model? I'm really curious to hear more about
that :-)

I see a number of people around me currently thinking about putting a "fair
salary" at the core of their future open-source work. Often it completely
changes what the delivered value should ultimately be etc (something I think
Mike did think about - I believe).

Do you have thoughts on that approach?

~~~
steveklabnik
My thoughts are mostly "I'm glad it's working out for Mike, and I hope others
can follow his path." I personally have taken pursued the patron model, but
that also is because with Rust, that's the only real way to do it.

~~~
thibaut_barrere
By patron model, to clarify, you mean a single large company paying most of
the bills and bringing funding, right? (either because they want to promote
innovation, or this generates financial gains, etc etc)?

I would also think that for a language, this is probably the best model (even
if it failed with IronRuby, or if JRuby funding sources moved over time etc).

This is a fairly large endeavour, so it makes sense (at least I believe it!).

~~~
steveklabnik
Yes. Though I mean me personally, not the language itself. I wanted to work on
Rust full time, Mozilla wanted to hire someone to work on Rust full time, it
fits.

~~~
thibaut_barrere
It's great that it works for you one on one, sincerely! I hope though that we
will see more occurrences of OSS projects where N customers are sending money
to directly finance the sustainability of the OSS product, just because it
delivers value for them (not a moral obligation or PR etc - note that I'm not
stating that Mozilla is doing this!).

I feel for libraries & components (unlike languages maybe, as we said), at
that specific scale, it can work very well.

~~~
steveklabnik
I do too.

------
steveklabnik
Hi all! Happy to answer any questions about this effort. I'm targeting 1.26.0
sometime in the next couple of weeks.

~~~
sanderjd
Thanks for all your hard work on this over the years!

Something I've wondered for awhile: what are the benefits of continuing Resque
development, rather than focusing on making the migration path to Sidekiq as
painless as possible? Put another way, what do you hear from users of Resque
(like Shopify, as you mentioned in another comment) as the reason that they
don't want to migrate off of it?

I was curious about this way back in 2012 when you announced work on Resque
2.0, and I had already made the switch to Sidekiq. I thought there must be a
good reason to keep working on it, but I wasn't sure what it was, and I'm
still not sure, so I figured I'd just ask.

~~~
steveklabnik
Sidekiq has a lot of attention amongst new developers, but has at least an
order of magnitude less users, at least, if gem downloads are something to go
by. So from my standpoint, those people deserve bugfixes, and "throw away what
works just to transition to something new" isn't really a great response.

Threading vs process-based jobs is a tradeoff. Threads are not universally
better.

~~~
mperham
As of today, Resque has 6,308,611 downloads. Sidekiq has 5,669,265 downloads.
Can you explain why you think "order of magnitude"?

~~~
steveklabnik
Hey Mike! Sorry, I 1) mis-read that number, 2) when we used to release more
often, the download counts for latest versions used to be 10x yours. I don't
think that's true at this point. I will update that cache in my brain and not
say this again, I apologize.

~~~
mperham
No worries, I just wanted to make sure you didn't have access to super secret
Ruby usage numbers that I didn't!

~~~
steveklabnik
I wish. :)

------
avitzurel
I have to be honest here. My initial thought when reading this a couple of
weeks back was that some things are better left to dead.

Resque has been an example of how not to run an open source project for a
while now. (3+ years).

We were working with Resque for a long while and had a lot of issues with
performance.

Sidekiq is awesome and it's being well managed and releases are frequent.

But, after thinking about it, I think that Resque has it's place in the
community, I definitely think that more than a single project doing this task
is needed.

Many people are already familiar with Resque and trust it, those people should
be taken care of.

I wish everyone involved a lot of luck with it.

~~~
sanderjd
This is another good answer to the question[0] I just posed to Steve. You're
right that competition is generally good. On the other hand, it would
(arguably?) be much more "Rails-y" to have a single conventional solution that
works well and gets a lot of eyeballs. Neither Resque nor Sidekiq are that
thing though. It has always struck me as a bit odd that Rails outsources the
decision on this component, which nearly every Rails project needs. I think
previously it was probably to avoid an external dependency (on Redis), but
perhaps that could be reconsidered now that >5.0 has eschewed that avoidance.

Having said all that, one of the things that is most exciting to me about
Elixir/Phoenix is that this sort of thing doesn't even come up, because a
great and mature solution is built directly into the runtime.

[0]:
[https://news.ycombinator.com/item?id=10911600](https://news.ycombinator.com/item?id=10911600)

~~~
steveklabnik
Newer Rails has ActiveJob:
[https://github.com/rails/rails/tree/master/activejob](https://github.com/rails/rails/tree/master/activejob)

------
AznHisoka
When i started my project, I used Resque but almost immediately switched to
Sidekiq a few months later. I understand it's almost impossible to run into
thread safety issues with Resque, but if you're doing anything CPU intensive,
Sidekiq really offers much higher performance. The maintainer also makes a
living off Sidekiq, selling it to businesses and enterprises too so you can
expect more frequent updates.

~~~
otterley
> but if you're doing anything CPU intensive, Sidekiq really offers much
> higher performance

The threading model (LWPs vs. forking) should have no impact whatsoever on how
quickly a CPU-intensive task completes.

~~~
andrewvc
Unless it's a short-lived CPU intensive task, in which case in matters a great
deal.

~~~
otterley
The overhead of fork() is negligible, well under 0.1ms.

~~~
andrewvc
There's also the issue, in ruby, that CoW doesn't really work due to the fact
that the GC dirties all pages.

~~~
mperham
This was fixed in Ruby 2.0.

------
fphilipe
The "again" in the title confused me. I knew that I read an article along
these lines a while back so I thought this is yet another one. Turns out it
was this post, which is dated October 23, 2015.

~~~
polysaturate
Aw poo...missed the date when I read it, and haven't seen it before. I'll
update the title.

------
whatupdave
I almost alway use Que [1] now. It uses postgres advisory locks instead of
Redis so one less dependency to worry about. You probably only need Redis if
you're running a tonne of jobs.

[1] [https://github.com/chanks/que](https://github.com/chanks/que)

~~~
callmeed
Rails 5 will require Redis so in the future it will be a dependency anyway ...

~~~
djur
Rails 5 does not require Redis unless you want to use Action Cable.

------
bdcravens
Interesting to note that the current stable branch has commits a couple of
days old.

~~~
steveklabnik
Yes, I've been doing some work, and so have some other contributors. Shopify
is upstreaming some patches so they can get back on master.

