

Celery 2.3 released - timf
http://www.celeryproject.org/news/celery-23-released/

======
jws
What is Celery?

 _Celery is an asynchronous task queue/job queue based on distributed message
passing. It is focused on real-time operation, but supports scheduling as
well._

 _The execution units, called tasks, are executed concurrently on a single or
more worker servers using multiprocessing, Eventlet, or gevent. Tasks can
execute asynchronously (in the background) or synchronously (wait until
ready)._

~~~
StavrosK
Ask described it to me as "a library for discovering bugs in the
multiprocessing library".

~~~
asksol
Yes, but note that multiprocessing rocks. It's just that celery uses it so
extensively :) (and now also eventlet/gevent if that suits better)

~~~
jnoller
And someone here has commit privs to fix those bugs too :) (ask does)

------
bryanh
Maybe it is just me, but I can never bring myself to setting up Celery and
RabbitMQ and all for some Django apps that could sorely use it. There seems to
be so many moving parts, last I checked there were like 3-4 different services
to tie everything together...

Anyone know of a simpler system or technique for achieving the same thing? Or
perhaps, a better way to go about using Celery.

~~~
dtran
IMHO, setting up RabbitMQ and Celery, especially with django-celery, ends up
being much simpler and less painful than rolling any kind of custom
queue/async processing. In terms of moving parts, for most setups, you can put
Celery and RabbitMQ under supervisord and almost never have to worry about
them.

If you don't need all the features of RabbitMQ or just don't want to deal with
it, you can use any of the other broker backends: Redis (if you don't need
persistence) or even with your default DB through the Django ORM.

~~~
Pewpewarrows
Fyi, redis has an appendonly option to provide persistence:

<http://redis.io/topics/persistence>

~~~
asksol
Yes, but I think what he was referring to was message acknowledgements, which
Redis doesn't have. If a worker reserves a message, and is abruptly killed,
then the message is lost.

------
Atilla
One of the best releases so far!

------
dtran
Great work as always asksol!

------
thedangler
I have no idea when I would need to use it. Can someone give examples of when
I might want to think of using it?

~~~
enjo
We use it extensively, essentially for anything that doesn't need to happen
"just this second". It's also replaced everything we used to do in cron, as it
easy to carry around in source control.

Most of our emails, calculating expensive things (we have some match-making
algorithms that run as tasks), sending notifications and all sorts of fun
things are all done in tasks. By moving as much of our code into celery as
possible it's been much easier to scale (adding a new node is trivial) and our
pages are much more responsive.

That Celery makes it all so very easy to do is icing on the cake.

~~~
notaddicted
I've set it up for a project where incoming HTTP requests can trigger outgoing
HTTP requests, something I found useful was a wrapper function that attempts
to execute the task asynchronously (the_task.apply_async) but falls back on
running it in-process (the_task.apply). This way if RabbitMQ isn't running
everything still basically works. The downside is that if RabbitMQ is running
but celeryd isn't, all the tasks just pile up in the queue.

------
guildchatter
Anybody here use Celery on their production system? Is it reliable/stable? Any
weird problems?

~~~
matclayton
We run it for mixcloud.com its awesome, saved us hundreds of hours of dev time
easily.

------
vae77
Great news! Awesome software!

