

Announcing DJJob, a PHP port of delayed_job - ericwaller
http://seatgeek.com/blog/dev/djjob-a-php-port-of-delayed_job

======
xal
I'm the author of Delayed Job. The popularity of it really surprises me to no
end, and always brings a smile to my face. DJ took about 15 minutes to code up
originally and is a product of one of those software rage moments ("why don't
people queue like this??!").

I put it on GitHub because Jeremy Kemper really wanted to have a look at it. I
thought the idea behind it was so naive that it would have occurred to
everyone who ever needed a queue and had discarded the approach for some
reason.

Little did I know that it was a fairly novel approach and that it would become
one of my most popular libraries, right after ActiveMerchant and Liquid.

Thank you for the PHP port!

~~~
ericwaller
Great to hear from you. I remember being impressed by how short and
straightforward delayed_job was when I first explored job queues. That's
actually one of the only reasons I even considered building/porting something
myself.

You're welcome, and thank you for delayed_job!

------
dkoch
Also the php PECL gearman driver (<http://pecl.php.net/package/gearman>) works
well with the C-based gearmand server (<http://gearman.org/>). We do some
asynchronous image processing with PHP gearman workers, and it's been a very
stable solution for us.

~~~
nphase
Gearman is especially handy if you expose your application to it. I've got a
wrapper that allows the app to push any function call out to the workers for
async handling. So your image processing (or whatever heavy/blocking task) can
be dispatched from within your standard libraries without having to maintain a
separate set of code.

------
thibaut_barrere
If you like Resque, too:

<http://github.com/chrisboulton/php-resque>

------
bl4k
you need to strip out the implementation specific classes such as propel and
the mysql-specific calls to make this useful to anybody else

for eg. an in-memory sqlite db would be ideal for this purpose (and a lot
faster)

~~~
ericwaller
Absolutely, dropping Propel is TODO #1. I just figured I'd throw it on github
and see if there was any interest first.

------
happybuy
In your example, does the call:

$worker = new DJWorker($options); $worker->start();

Actually: 1\. block until all jobs stored in the database have been completed
2\. simply run a single job and return 3\. return immediately and run all the
jobs on a separate thread in the background

I'm wondering where in a web application would be the best place to call
$worker->start()?

~~~
aonic
I think the point is to run the worker outside of a web request and using a
cron job

~~~
andrewjshults
Or using the command line version of PHP and GNU Screen (I haven't fully
studied the code, but it looks similar to a queue system that I hacked
together a couple of days ago where the worker is wrapped in a while(TRUE)
loop).

In the interest of some outside figures, how have you found this to scale? I
implemented a similar solution as more of a stopgap until I had the time to
get RabbitMQ installed with a "proper" worker dispatcher (more moving parts
but less stress on the DB). I mostly ask because we implemented it for our
notification system where we had to generate a send large batches of custom
emails which was bogging down the page (and hard to debug where in the chain
we were getting failed messages).

~~~
aonic
Not sure who your question was directed to, but I've implemented a similar
system in the past for a website with 10m+ users and 15+ web servers. It
scaled just fine, we were sending close to 150K emails a day, among other
background jobs for reporting, data cleanup, automated spam checks, etc

------
mikeytown2
Made something very similar to this in PHP about a year ago. It's buried in
the Drupal project called Boost. It's the multi process crawler. It supports
up to 8 processes crawling your site getting the cache warm. Gets call on cron
& then 15 seconds after drupal's cron finished it starts up all 8 "threads".
I've used it for image processing as well. Very stable will run for days if
you give it a big enough queue (millions of pages to crawl).
<http://drupal.org/project/boost>

