

Ask YC: Which queue/background worker solution? - andr

I want to distribute tasks across several servers and processes. I've found several solutions, but couldn't find any useful third party reviews. So can anyone recommend a good solution from their own experience?<p>Currently, I'm looking at:<p>- Scarling/Kestrel http://github.com/robey/kestrel/tree/master<p>- Gearman http://www.danga.com/gearman/<p>- RabbitMQ or any other AMQP solution (but that looks as overkill)<p>- rolling my own with REST or MySQL
======
RobGR
I have done this via writing my own system.

At one workplace I had perl scripts that ran on every Unix machine there at
night (about 60 Sun's on engineer's desktops) that checked with a central
repository, locked a task, and downloaded it and ran it and filed the results.
I have since done the same with a mysql "task" database, and the client
checked in and picked up a task and ran it.

In both cases the system was not general and specialized to the particular
jobs at hand.

It really depends on what type of tasks you have, and what resources you have
in terms of computers. Without those details, the best I can say is that
rolling your own is not that hard, and you should not be intimidated by it.

