
A Simple Pattern for Jobs and Crons on AWS - douglasfshearer
https://blog.skyliner.io/a-simple-pattern-for-jobs-and-crons-on-aws-2f965e43932f
======
Zombieball
An alternative would be to just use AWS lambda:

> You can create a Lambda function and direct AWS Lambda to execute it on a
> regular schedule. You can specify a fixed rate (for example, execute a
> Lambda function every hour or 15 minutes), or you can specify a Cron
> expression.

[http://docs.aws.amazon.com/lambda/latest/dg/with-
scheduled-e...](http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-
events.html)

It would remove the need of infrastructure for "task workers". Although,
depending on your stack / setup, it of course may be easier to have your
application tier double as a task worker due to complicated dependencies or
startup times.

Additionally, I am not sure if everyone is aware but SQS does offer FIFO
delivery with guaranteed once-only "processing". There are some interesting
caveats to this, definitely worth looking into though :)

~~~
coredog64
Yes, but...

AWS Lambda has a limitation on the number of functions you can attach to a
scheduled event. What this means is that you wind up with a shitload of 'every
5 minutes' schedulers that all need unique names. WTF Amazon?

~~~
idbehold
Hook up just one Lambda function to the scheduled event. And have it invoke
other Lambda functions.

------
luhn
I've used SQS for a simple job queue before. It works well and it's nice not
to have to worry about more infrastructure. To make things even better, Celery
now has an SQS broker, which I'd like to try out.

I've also had success with an HTTP subscription in SNS for simple async jobs.
Having your tasks be just another HTTP endpoint is really convenient. It's not
actually a task queue, but it's often a good-enough substitute.

One frustration I've had with it is that Amazon signs SNS messages, but I
can't find a Python library for validating the signature. Not even AWS's
flagship API client (boto3) has the functionality.

~~~
Ysx
> [...] I can't find a Python library for validating the signature. Not even
> AWS's flagship API client (boto3) has the functionality.

I wrote validatesns [1] for this, give it a try!

(Not sure why there's an official PHP validator but not Python...)

[1]
[https://github.com/nathforge/validatesns](https://github.com/nathforge/validatesns)

------
revicon
As simple as this article is trying to make things, it still feels like it's
trying to re-create what elastic beanstalk workers already do out of the
box...
[http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-...](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-
features-managing-env-tiers.html)

------
krallja
You could also use `sqsd` (an Elastic Beanstalk worker environment) and avoid
having to write any queue reading code.

------
nodesocket
Can't wait until Google Functions[1] adds the ability to schedule and define
recurring function calls.

[1] -
[https://cloud.google.com/functions/](https://cloud.google.com/functions/)

------
WaxProlix
Something similar, and perhaps even simpler: Make a few SNS topics that
lambdas can subscribe to - either one per method or one per method type, and
then filter more in code - and push notifications to those from your service
layer as needed to invoke async workflows. I've been doing this recently and
it's pretty pain-free.

Been meaning to look at Step Functions for a lot of this async work stuff, but
haven't had time yet (plus it's nice to let these things mature a bit before
diving in).

------
glumpyfish
For jobs that take longer than Lambda will allow, I use Datapipeline to run a
ShellCommandActivity, which can run a bash script on an EC2 instance.

~~~
bdcravens
Running an ECS task from Lambda is another option.

