
Large Problems in Django, Mostly Solved: Delayed Execution - mnemonik
http://ericholscher.com/blog/2010/jun/23/large-problems-django-mostly-solved-delayed-execut/
======
jonasvp
Celery together with RabbitMQ (or another message broker) is great for
big/huge projects. For basic job queueing and delayed job execution,
beanstalkd (<http://kr.github.com/beanstalkd/>) is in most cases a better
choice. It takes almost no memory, no configuration, and just works.

I wrote a small library for Django that makes writing jobs for beanstalkd
easier (<http://github.com/jonasvp/django-beanstalkd>). There's also a
management command for starting (threaded) workers. Makes writing and calling
jobs from Django very simple.

~~~
asksol
Could you please elaborate _why_ it is a better choice? I don't see how
django-beanstalkd could take much less memory than celeryd. If it's about the
broker (beanstalk), you could have implemented a beanstalk backend for celery
as well, then you would have had a tested implementation of the worker part.

Btw, If you would be interested in writing a beanstalkd backend for celery I
would be happy to help you out!

~~~
jonasvp
Good point - I was under the impression that celery _only_ works with an AMQP
message broker. Guess I should have read the description more closely before
starting on django-beanstalkd.

I'm happy with my lightweight model for now but I'll definitely look into
celery again, when the need arises.

------
jnoller
I can personally attest to the fact Celery is pretty awesome, and a nice
compliment to your Django app. The author - Ask is also doing some awesome
things with multiprocessing in general - check out
<http://pypi.python.org/pypi/billiard/>

------
tomazmuraus
I use Celery myself as well and it's awesome.

It supports multiple messaging queues and backends for storing results
(RabbitMQ, MongoDB, AMQP, Redis, ... - I personally use Redis, because I don't
need all the features and overhead from RabbitMQ) and also has a great
documentation and friendly support on IRC.

And if you watch the Celery project on github you can see that the author is
basically working and improving the project on daily basis.

------
pxlpshr
We've been using Celery with Django for a solid 8 months or so. It's been a
blessing for our needs, and extremely easy to scale.

------
daredevildave
How does celery differ in functionality than a message queue like RabbitMQ?

~~~
kingkilr
Celery leverages a message queue (including Rabbit, which is how we deploy
it), Celery provides an easy API to define tasks, push them onto your queue,
and run them on workers.

------
barnaby
Do scheduling tools in a web framework protect you from having to write
threading code yourself? Or is it for a completely separate kind of side-
process than when I would want to write threads in my web app code?

Also, is that like Quarts in the Spring Framework?

~~~
steveklabnik
> Or is it for a completely separate kind of side-process than when I would
> want to write threads in my web app code?

This isn't about threading, really. I'll explain it in terms of Rails's
delayed_job because I'm not familiar with the details of Celery:

    
    
        * Let's say I want to send an email, via the function Mailer.deliver_notice(email)
        * With delayed_job, I turn this call into Mailer.delay.deliver_notice(email). This puts an entry into a Jobs table in the database that makes note of what I want to run, with all the details.
        * In the background, I have a daemon running that checks the Jobs table every five seconds for jobs that need to be run. It uses the information in the table to execute the deliver_notice function.
        * That's it.
    

In many cases, the job will update some sort of field in your model that you
can query to see if it's been run yet or not.

------
nphase
I'm a big fan of doing this with Gearman (<http://gearman.org/>). Super simple
interface, very easy to shove stuff into the background.

------
ebtalley
It's a little off topic but I would like to take a moment and profess how much
I LOVE the fact that hacker news has so many articles on all languages,
popular or not.

------
jsdalton
What are the advantages/disadvantages of using a messaging framework like
Celery vs. handling asynchronous tasks in threads?

~~~
bfirsh
Celery workers can run on any networked computer - it can be scaled
infinitely!

