

Celery - Distributed Task Queue - albertzeyer
https://github.com/celery/celery

======
memset
Here is a question: I often find myself wanting to do background tasks within
my application (say, a Flask app) _without_ running celeryd. While celery is
quite useful, I just want to spin up an application and have it Just Work
without external dependencies. With celery, now I have to run celeryd, run
some sort of queue (redis, rabbitmq, etc), and just _worry about all of these
pieces_ simply to execute long-running background tasks.

Are there options for this? Akin to an sqlite for tasks, in python?

~~~
oxtopus
Running your Flask application with uWSGI could allow you to do just that by
way of the spool and other related decorators: [http://uwsgi-
docs.readthedocs.org/en/latest/PythonDecorators...](http://uwsgi-
docs.readthedocs.org/en/latest/PythonDecorators.html)

~~~
mh-
+1 for uWSGI. The Spooler is awesome and also, if you need lower-level stuff,
have a look at its SharedArea and Queue[1] frameworks, as well as Mules[2].

[1] [http://uwsgi-docs.readthedocs.org/en/latest/Queue.html](http://uwsgi-
docs.readthedocs.org/en/latest/Queue.html) [2] [http://uwsgi-
docs.readthedocs.org/en/latest/Mules.html](http://uwsgi-
docs.readthedocs.org/en/latest/Mules.html)

------
pepijndevos
What is there to see? Celery is great, but has been around for ages. Anything
new I'm missing?

------
nemothekid
I've been looking to replace our in-house postgres based task queue with
something more suited for the task. What has been turning me off about
celery/rq/qless is that they seem tied to the host languages (python/ruby),
our backend is Go, and the Postgres mostly acts as a datastore. I'd love a
somewhat dropin replacement where instead of reading/writing to pg, I'm
reading/writing to redis/some other in memory store.

What I want is the ability add a task with a priority and metadata, and be
able to pop those tasks in O(1) time.

------
ivankirigin
I think RQ is simpler and better. [http://python-rq.org/](http://python-
rq.org/)

Celery has a lot more control and features that you probably don't need.

~~~
the_cat_kittles
I have tried to use RQ a couple times as a result of this thinking. Each time
I end up going back to Celery because there always seems to be something I
need that RQ doesn't have. For instance, in the most recent project, I wanted
to execute tasks X minutes in the future- celery's eta kwarg is perfect for
that, and I'm not sure how you would do it with RQ.

------
omarish
We use celery pretty extensively. It's nice because you get the flexibility of
multiprocessing, but also the ability to distribute tasks across many
machines.

There is definitely an overhead to get it going. Using redis as your messaging
backend reduces this cost somewhat.. rabbitmq is overkill for most of the
cases we've used celery.

Have a look at rq if you're looking for something more lightweight:
[https://github.com/nvie/rq](https://github.com/nvie/rq)

------
rdtsc
A simple solution (and I see a couple of requests here) is to use eventlet as
the underlying concurrency library and then just spawn a green thread (or a
green pool! to limit concurrency) to handle long running/background tasks.

------
jensenbox
I have yet to read anywhere why using Postgres or Postgres with Django is a
bad idea. Everything says a vague "suboptimal" without any real backing of
that statement.

Anyone know the real reasons behind that statement?

~~~
gtaylor
The workers have to poll Postgres repeatedly at intervals, whereas a worker
using a message broker (like RabbitMQ) connects and just waits for something
to get tossed its way. Having your task broker separate from your database can
save a lot of DB IO if you've got a larger number of workers or a busy task
queue. RabbitMQ (and the other more full-featured message brokers) can also do
a lot of really slick things with routing, prioritization, and all sorts of
other goodies.

Of course, this doesn't matter if you're running a small site with low
traffic. I wouldn't get too caught up in worrying about broker selection
unless you're cranking out a lot of jobs, or have special needs.

~~~
jensenbox
So if I am reading this right, what you are saying is that RabbitMQ is more
'push' and a DB is more 'pull' or rather 'poll'.

Perhaps they should break the brokers down into those two groups.

I seem to recall Redis having a pubsub style interface, which makes it much
more push but still is a bit cloudy in my mind.

Thanks for the reply!

------
buster
I have been using Celery in a recent python project with rabbitmq and it as a
very good experience, celery works really nice and reliable. Would use it
again!

------
McGlockenshire
Outside of using AQMP, what does this get me over using Gearman?

~~~
gtaylor
If you're using Python and/or Django already and don't need to mix with other
languages, Celery is pretty easy to work with, has excellent documentation,
and is well-maintained. With that said, while there is no boundary to playing
nice with other languages, it seems like only recently have people started to
get serious about trying to do so with Celery (and sharing/supporting their
work publicly).

I haven't used Gearman personally, but if I had a polyglot project, I'd give
it a serious look.

------
guiomie
How is Celery different from RabbitMQ ?

~~~
alexmic
Celery is a task framework. RabbitMQ is a queue server. Celery can use
RabbitMQ as a message broker i.e a way to transfer the messages from your
application to the workers.

