

Async Task Queues - bluemoon
http://99designs.com/tech-blog/blog/2012/02/17/async-task-queues/

======
simonw
I think a task queue should be a default part of the web stack in exactly the
same way that memcached is - once you have one set up a huge range of problems
become things that you can quickly solve using a queue, which is good news for
performance, scalability and maintaining a clean architecture. The initial
setup costs pays off very quickly.

------
efsavage
Part of me sees things like this and my inner curmudgeon wakes up and yells
"This is what happens when you use things that are new but not better! This is
standard knowledge that has been around for 20 years! If you only used a
_real_ web language/stack you wouldn't be reinventing everything! JMS
anyone?!"

However, my better angels re-read it and say "Well, at some point you didn't
know all that, and this is actually a well-written intro to the topic".

So ... good article!

------
deet
Task queues also allow for interesting web application architecture when
combined with technologies that enable the server to push events to the client
(think WebSockets, Socket.IO, etc).

The potential is far greater than simple tasks like sending an email, fetching
external data, etc. If the server has the ability to push updates, you can
start to use task queues in core user interactions. This is possible when
workers can notify clients when their task is complete.

For example, you can have clients submit HTTP requests to the web server, the
web server dispatch jobs to the task queue, worker processes (potentially in a
different language than your web server!) retrieve those jobs, and then the
workers directly notify clients when the job is complete. Combined with
client-side logic, the result can be a good user experience, even when the
server is performing complex tasks.

For example, I'm using this strategy to write web-based statistics software
(think web-based Stata or SPSS). Without this type of asynchronous
architecture, it simply wouldn't be possible to provide a good user experience
because the core things the user wants to do are computationally intensive.

------
the_cat_kittles
I'm by no means versed in async task queues, but Celery
(<https://github.com/ask/celery>), a python task queue, has worked quite well
for me. Interesting article.

------
jbarham
Amazon's SQS (<http://aws.amazon.com/sqs/>) is a nice queue implementation
with a very simple API for widely distributed systems. See the boto Python AWS
library documentation for example usage:
<http://readthedocs.org/docs/boto/en/latest/sqs_tut.html>.

Google's App Engine task queues are also very easy to use as the workers are
just normal URL handlers in your app and the payload is standard HTTP data.
See Python documentation at
<http://code.google.com/appengine/docs/python/taskqueue/> and Go documentation
at <http://code.google.com/appengine/docs/go/taskqueue/>.

------
alexchamberlain
It would be really cool if we could compile a list of tasks on a webserver
that these task queues could carry out.

The article lists: Image resizing Sending emails Third Party API calls

What about? Database optimisation Static file compression

~~~
dhotson
Yes and Yes. We use async tasks at 99designs for all kinds of background
tasks. It's a surprisingly useful and versatile system for a variety of
things. It's a long list. :-)

Our rule of thumb is basically anything that's going to take more than ~100ms
we definitely don't want our users waiting for it in their page load time.

------
kellysutton
At LayerVault, we're using delayed_job
(<https://github.com/collectiveidea/delayed_job>) for our queuing system.
Everything goes through a database, so the performance won't be as good as
something in memory. Also, I feel safer with the workers only communicating
with queue servers and not with application servers. Just a litte more
abstraction :)

------
conroy
Storing the queue locally is great for speed, but it sacrifices durability.
When an app server crashes, all those queued tasks are lost in the ether.

~~~
dugmartin
The server they are using, beanstalkd, has a persistent bin log option. Of
course if your disk also crashes you are up a creek.

------
hcho
I once built a task queue using redis, start-stop-daemon and php. start-stop-
daemon can turn vanilla php scripts(or any kind of script for that matter) to
a worker.

