

Ask HN: What do you use for job scheduling? - remotecode

Hi all,<p>I’m looking for a good job scheduler solution that fits my use case. In the past I have used the following:<p><pre><code>  - Cron
  - SOS JobScheduler (http:&#x2F;&#x2F;www.sos-berlin.com&#x2F;modules&#x2F;cjaycontent&#x2F;index.php?id=osource_scheduler_introduction_en.htm)
  - Celery with Celery Beat
</code></pre>
All of these have pros and cons. None of these have satisfied the requirements that I have below, so I find myself building workarounds. SOS JobScheduler is the closest to meeting my requirements.<p>I’m curious what the standard tools for job scheduling are that people are using in production?<p>Here are my requirements:<p><pre><code>  - Start a job at a given time
  - Start a job at a given interval (e.g. every 15 minutes)
  - Start a job on event:
    - Successful completion of prior job (e.g. a dependency)
    - Successful completion of multiple prior jobs
    - Failure of prior job
    - Keep all stderr and stdout for review
    - Track job run time (e.g. are my jobs getting slower?)
    - Alert on failure
    - Retry job
</code></pre>
I would love to hear from the HN community what technologies you are using and how you like them &#x2F; pros and cons.
======
dilithiumhe3
I would recommend using/looking at Amazon SQS. It seems to all (or atleast
most) of your use cases and is managed so you won't need to deal with yet
another system. It is strictly a message queue so you would still need to do
some work to get things wired up but it should be straight forward.

Another mention, Amazon Simple Workflow is also something you may want to look
at. I haven't used it in any project but seems like it could, sort-of work
here. (not particularly sure though).

------
rajacombinator
Which ones of those use cases does Celery not fulfill? I currently use cron,
which is pretty bad, and have an eye to move to Celery in the future. (Because
it seems legit.) Will be pretty disappointed if it can't handle those reqs as
they seem pretty standard for a job scheduler.

