Ask HN: What do you use Celery / RabbitMQ for? - zabana
======
malux85
To easily distribute jobs over a cluster.

Running right now:

I have a database of 100,000,000 molecules, and I need to compute the force
field on all of them. So I put all of the IDs in the celery queue, and then
start a worker across 6 nodes,

The job pulls the row from the database, computes the force field, and then
stores the result.

Lots of jobs like this: \- I used to use it to tune hyperparameters of shallow
models \- Used it to batch convert a bunch of compressed files from one format
to another \- Use it with celery beat to schedule scraping of URLs

------
gerenuk
We have written distributed crawler using celery/Rabbitmq.

Analyzing social signals real time and it is processing more than 10 million
requests everyday. On the way to scale it to process more than 50-100 million
requests per day.

Other than that doing nlp tasks along with machine learning using topic
modeling, ner etc.

