
Show HN: goworker – a faster, Resque-compatible background worker - benmanns
http://www.goworker.org/
======
benmanns
This is a weekend project I started on as a way to learn Go that I quickly
realized would be something worth sharing. It lets you queue jobs from Rails
that are processed in Go, which can speed things up considerably.

I would appreciate all kinds of feedback, and for anyone who wants to try
goworker out, if you give me your email at the bottom of the page, I'll send
you my contact info for some free one-on-one help getting started.

~~~
atonse
Very cool - I just interviewed with a company that is using Sidekiq to process
up to millions of jobs (at about 2/sec per worker) and I asked them to look
into using Go for workers. I can send this to them.

~~~
benmanns
Awesome! Can you send me your contact info? Mine is in my HN profile.

------
Titanous
I have a similar project that we use in production; a Sidekiq clone in Go:
[https://github.com/cupcake/gokiq](https://github.com/cupcake/gokiq)

I should really document it. It has full type safety, stats, scheduled jobs,
error reporting, retries, etc.

~~~
benmanns
That's really cool. Where are you using it in production?

Full Sidekiq compatibility (scheduling, retries) was my next goal.

~~~
Titanous
It is part of the infrastructure for Cupcake's Tent hosting service
([https://cupcake.io](https://cupcake.io)).

------
steveklabnik
Hello there! Resque maintainer here. This is super neat! I'll make sure to
link to goworker in our docs. I have wanted to do something like this, but
haven't gotten around to it yet.

Please let me know how I can help you stay abreast of what we're doing with
Resque 2.

~~~
benmanns
Hello! Thanks for your comment and tweet. I've been watching both Resque and
Resque 2 while writing goworker. I would definitely like to stay in contact
with you.

------
jemeshsu
Interestingly there is a Sidekiq compatible job queue written in Go that has a
similar name: go-workers [https://github.com/jrallison/go-
workers](https://github.com/jrallison/go-workers)

Wondering how these ruby/rails inspired work queues compare with NSQ.

~~~
jonnii
I'm using this and it works great. We even managed to get it deploy to heroku,
which was pretty magic.

~~~
benmanns
How do you handle when Heroku sends a SIGTERM to kill your process? I couldn't
find a way to preempt running workers, so everything running on Heroku has to
finish within 10 seconds or you can lose it.

~~~
jrallison
Hey Ben,

Author of go-workers here. On SIGTERM, I stop accepting new work, and wait for
all running workers to finish before halting.

If workers take longer than the 10 seconds Heroku gives you, go-workers uses
reliable queueing (using
[http://redis.io/commands/brpoplpush](http://redis.io/commands/brpoplpush)) so
the job will run again next time you start up the process.

~~~
benmanns
Okay, cool. I do the same thing on the polling side, but don't use reliable
queueing yet. I think that is probably the best way to handle the failures.

------
AznHisoka
I took a look, and am a little confused. Does the code for the workers need to
be written in Go, not Ruby?

Have you also tested the performance of this with 40-50 workers doing
intensive I/O work such as crawling web pages? One of the main disadvantages
of Sidekiq is that the workers just freeze every 15-30 minutes when you have a
ton of workers crawling web pages. The only workaround for me is to setup a
cron job that restarts the workers if I detect this pattern.

~~~
benmanns
Yes, workers are written in Go, but can be queued from Ruby. I'll put together
an example in the goworker-examples [0] to test web crawling.

[0] [https://github.com/benmanns/goworker-
examples](https://github.com/benmanns/goworker-examples)

~~~
dkhenry
So workers written in go are 1000x times faster then workers written in Ruby?
It almost seemed from the documentation that the workers were written in Ruby
as well.

~~~
benmanns
They are typically 10x faster in "raw" speed and 100x faster for concurrency.
However, since Go's goroutines take almost no memory, you can run over 1,000
workers in the same amount of memory as one Ruby worker.

On the queuing side, you use Ruby. For the examples, I was trying to show what
the "equivalent" Ruby worker was so you could get an idea of what kind of
workload it was. However, I do see that it is confusing.

------
skion
And so the age starts where we rewrite in Go for performance all cool things
Ruby, Python...

~~~
agentultra
You also forget the competition from Javascript as well. Rewrite all of the
things in all of the languages.

I'd much rather see neat algorithms and libraries being re-written in C with
an FFI-friendly API. I think it would reduce the number of "Show HN: I rewrote
X in Y" posts. The majority of HN users rarely care about Y. Such posts are
just attention-grabbing noise.

If it was written in C you could just write bindings to it in Y and nobody but
Y developers would need to hear about it. Instead we could read about X on HN.
X is interesting.

~~~
iamwil
Actually, I've wondered if there's a docker for libraries in different
languages. The closest I've found are http APIs at the application/service
level, or libraries mature enough to be run at the command line, so you can
use them in conjunction with other unix commands.

It doesn't seem right that we can't leverage libraries in other languages
without significant effort.

------
adefa
There is Java implementation of Resque as well:
[https://github.com/gresrun/jesque](https://github.com/gresrun/jesque)

I've been using jesque to consume Resque tasks with no problems, and
performance is great.

~~~
benmanns
Cool! I'll check Jesque out and add it to the benchmarks.

~~~
mikeni
interested in your findings

regarding goworker, so the worker logic has to be written in go correct? for
my rails app if its something simple like sending email, I can use that, but
if it involves something where I need activerecord is it still a good option?

------
jrochkind1
> _For simple dequeuing, deserialization, execution, and updating stats (as
> with the Hello worker above), goworker is 80x faster than Resque and 23x
> faster than Sidekiq._

In real world use cases, how often is that kind of overhead a significant
portion of total run time, compared with the actual work? If it's a small
portion, would that make the speed up fairly irrelevant, does it actually
speed up real world use cases significantly?

Of course, your actual work payload may be faster too in go than ruby if you
know how to write it well.

~~~
sanderjd
I would actually expect the simple dequeue-deserialize-update-something-in-
redis use case to be the "worst case" comparison and that the more work the go
worker does, the better the performance numbers would look.

It seems like using something designed from the start for cross-technology
communication (like 0mq) may be a better idea, but I suppose there is already
lots and lots of existing resque client code out there.

~~~
steveklabnik
Since jobs are just encoded as JSON into Redis, it's pretty already cross-
technology compatible. Furthermore, we're committing to a format specifically
for Resque 2.0, so you can guarantee that you'll be interoperable.

~~~
sanderjd
Neat. I really like the idea of Resque being able to be thought of as a
protocol specification plus a reference client and server implementation.

------
joevandyk
How would you write integration tests if you had a rails app that pushed jobs
out to goworker?

Say a user clicks a link that creates an async job that does a API call
somewhere. What's a good way to test that entire process at once? You can
easily test them separately, that clicking a link creates an async job, and
that the worker can process the job, but it's useful to test the whole system
at once.

~~~
benmanns
That's a great question that I haven't thought about before. You could
possibly write a Go wrapper that executes your workers, getting the job from a
command line argument rather than Redis.
[https://gist.github.com/benmanns/6584142](https://gist.github.com/benmanns/6584142)

~~~
joevandyk
The test should use redis though, to test the whole stack.

Same question goes for any time of queue system, like quque_classic, which
stores the jobs in a postgresql table.

------
kohanz
In my side-project, which is my first foray into Rails, I use a Resque
background worker to do processing and it takes a significant amount of time.
As I've heard more about Go, I've wondered how I could use it to speed up this
background process. Here is my answer! Thank you very much, I'm looking
forward to trying this out as it fits my needs exactly.

~~~
benmanns
Awesome! Contact me (info in HN profile), and I'll help you get started.

------
g5pw
A nice log graph would be better than the linear ones IMO, especially when the
scale stretches a lot (like in the Database Insert tab)

~~~
benmanns
True, but I thought that that would look disingenuous, because it would look
like the worker times scale logarithmically when they are about as linear as
you can get.

------
hmottestad
Please label your graphs. X-axis, Y-axis and title :)

~~~
benmanns
Ah, yes. I'm using Flot and was having trouble getting good looking labels.
I'll work on that right away, but in the meantime: X-axis is jobs, Y-axis is
seconds to execute.

~~~
dylz
Might I recommend highcharts? It's a lot easier to work with than flot IMO

