

HireFire - The Heroku Worker Manager - meskyanichi
https://github.com/meskyanichi/hirefire

======
KarlFreeman
Very handy. Was looking at using <http://www.simpleworker.com/> as an
alternative to the uncost effective Heroku DJ's but hopefully HireFire will
sway me back to using Heroku.

~~~
meskyanichi
Hey. I was actually also using SimpleWorker prior to HireFire, but I find that
setting it up is a bit hard / takes long since you have to set isolated code /
credentials / etc to their servers in order to process background jobs. Also,
the jobs aren't guaranteed to run immediately after they are queued, may take
up to 10 minutes on the cheapest plan. When deployed to Heroku it's easy since
Heroku will just use the same compiled slug it uses for Dyno's, no additional
code/configuration required, it just works. I also found that on SimpleWorker
a job that took only 3 seconds on Heroku, would run for 30-90 seconds on
SimpleWorker (no idea why).

That said. There are some advantages when using SimpleWorker such as the web
interface, scheduling recurring jobs, and other useful options. Especially
when you aren't running on Heroku, and memory is an issue if you have a small
VPS. In that case, offloading background jobs to SimpleWorker is ideal. But I
find that HireFire will make Heroku Workers cheaper than SimpleWorker due to
the fact that jobs appear to take (a lot) longer to finish on SimpleWorker
than on Heroku (probably due to the start up time? I'm not sure why).

~~~
KarlFreeman
Couldn't agree more. The setup time of SimpleWorker is alot ( especially when
dealing with specific gems ) and the fact that for DB work you have to create
a separate non concurrent connection to the DB per job makes those times you
talk about a reality. Ok, you could use the 'expensive jobs' but Heroku's DJ's
win hands down in all tests I've done. Plus who doesn't like the
handle_asynchronously :function.

~~~
meskyanichi
Exactly. :) The DB connection is also quite frustrating depending on the
location of the DB. MongoDB with MongoHQ or MongoMachine is fine since it's
external already. But otherwise you have to know how to configure for example
your MySQL PostgreSQL instances to allow incoming connections from external
locations, and also becomes potentially insecure depending on your sysadmin
skills.

I find Heroku + DJ + HireFire so much simpler, either using that
handle_asynchronous method, or just called delay.my_method(*args) is
brilliant. No need to think about isolated code or credentials due to the
environment variables that are already set on Heroku.

I should look in to integrating Navvy and Resque as well.

~~~
chadarimura
Hey Guys, I'm a co-creator/founder of SimpleWorker.com. Can't say enough how
appreciative we are of the feedback both good and bad. We've been experiencing
pretty strong growth and through this are identifying the sticking points and
working through them as fast as we can.

The isolated code issue has been a big issue, and we're addressing this
through calls like merge_gem(), and soon merge_mailer(). It's a challenging
task, and we see the gaps, but we're improving every day.

Runtime: this is also a top priority. We're working with each and every
customer to ensure that their workers execute quickly. 3 seconds vs 30-60 is
totally unacceptable and we won't leave you hanging here.

Queue speed: we're prototyping alternatives to http queuing which will
dramatically speed up queue times.

This is just the tip of the iceberg for roadmap... Great work on HireFire. I
see cases for both. Potentially massive parallelization on SimpleWorker
(hundreds if not thousands of concurrent jobs), advanced scheduling needs,
etc.

Anyways - we're listening - so any and all feedback is well received. If
anybody has needs outside HireFire, in conjunction with, or otherwise, shoot
me a line and we'll get you up and running at no cost to start.

Chad Arimura chad[at]simpleworker.com

<http://www.simpleworker.com>

~~~
meskyanichi
Hey Chad, great to hear! I also (despite having created HireFire) still have
use-cases for SimpleWorker, however in my case they will likely be for my VPS'
rather than on Heroku. On Heroku (even though there isn't a UI, unless you use
Resque), it's already a partially managed service to host workers, it runs
from the same compiled slug as the dyno's use so all the credentials / env
variables are all shared among dyno's and workers, no need to re-apply
configuration and such.

I think for me the hardest part was getting it to work the first time, waiting
quite a while for the jobs to run to see if I managed to get it to work. The
merge_gem is a huge improvement, though, handling credentials and re-
connecting databases and other things are quite a hassle to set up (imo). I
have to say that once it's all set up, and if the 30-60 start-up issue is
gone, that it does seems like a beast for processing a lot of jobs
concurrently and is exciting.

I'm definitely still looking forward to seeing future progress of
SimpleWorker. So please do keep up the good work!

------
freerobby
Nice work, glad to see heroku API gems getting some love.

I wrote a similar gem back in January and have had good luck with it:
<https://github.com/freerobby/middle_management>

It's fully configurable via environment variables, so you can make changes
without redeploying. It also lets you specify a minimum number of workers
(even if there are zero jobs) so that you don't have to wait for workers to
spin up if you don't want to.

~~~
meskyanichi
Thanks. I like your approach of not having to deploy to change the min/max
value of workers. Though it's hard to implement (I think) if you're working
with a worker/ratio configuration like HireFire. Currently HireFire also
supports the Mongoid ODM. I'm looking in to implementing the Resque worker as
well so people can use HireFire with Resque with Redis on Heroku as well.

Thanks for your input!

~~~
freerobby
If I could fork your code and get it working with environment variables, would
you be interested in pulling that in? (Not sure if you made a deliberate
decision to avoid that).

MiddleManagement works great for my needs, but if these libraries serve the
same use case(s), I'd rather converge towards one library than have things
spread out.

One other thing you may want to add is a retry loop for the heroku API calls.
Every now and then, the heroku gem will throw a
RestClient::ServiceUnavailable: 503 Service Unavailable exception. In the
worst case, I'm not sure if this is best handled in the gem or at the
application layer; however it's definitely worth retrying a couple times if
you can avoid the blowup completely. I got some feedback from the Boston.rb
group on handling this if you're interested:
[http://groups.google.com/group/boston-
rubygroup/browse_threa...](http://groups.google.com/group/boston-
rubygroup/browse_thread/thread/ede9fa007735971/9d586498680f958f?lnk=gst&q=robby%40freerobby.com#9d586498680f958f)

~~~
meskyanichi
Thanks for the heads up. Seems like a good idea to catch the 503 errors. Also,
for now I'll leave all the configuration defined in pure Ruby. Maybe
eventually I'll consider using ENV var's to handle that.

Thanks for your feedback!

------
nestlequ1k
Looks great. Hoping to see resque support on it soon.

One thing this really shows the need for is a real api on heroku. Having to
hardcode your username/pass into env variables is pretty ghetto. Hopefully
heroku devs are watching and listening. Oauth plz??

~~~
meskyanichi
Thanks! Looking at Resque and will probably be implementing it fairly soon.

Ha! And yeah, the ENV variables. :)

------
gerrit
For the low-traffic scenario there's also an older, much simpler solution:
<https://github.com/mtravers/cheepnis>

~~~
meskyanichi
Interesting and light-weight approach. Though, does it support other ORM's or
ODM's like Mongoid? Currently HireFire supports ActiveRecord and Mongoid with
Delayed Job. Planning to add Resque with Redis support fairly soon.

------
anulman
I was looking for something like this months ago. Thanks for building, and I'm
looking forward to (at least trying to) contribute!

------
mrdigsus
Looks extremely cool. Can't wait to kick the tires.

------
ddemchuk
Does anyone know of gems like this that will work on standard VPS servers? I'm
launching something that completely depends on Delayed Job for processing data
and would love an interface to manage queues and number of workers spun up at
a given time. Also, visually watching the queue (a la resque) would be
amazing.

The best I've found so far is:
[http://trevorturk.com/2011/01/21/delayed_job_admin-a-
simple-...](http://trevorturk.com/2011/01/21/delayed_job_admin-a-simple-admin-
interface-for-delayedjob/)

~~~
meskyanichi
Hey. The gem actually has a local environment option which makes it work on a
VPS similar to how it works on Heroku. You could try that and see how it
works.

[https://github.com/meskyanichi/hirefire/blob/develop/lib/hir...](https://github.com/meskyanichi/hirefire/blob/develop/lib/hirefire/environment/local.rb)

Also, as far as I know that DJ Admin Interface is the only one for DJ that
currently exists. If you want to use Resque, I'll be implementing Resque for
HireFire soon.

