
Ask HN: What job scheduler do you use? - pmchorus
In a previous company, I used the in-house job scheduler. Despite some bugs and lack of documentation it did what we asked for: schedule jobs according schedules and timezones, and handle dependencies.<p>Now I&#x27;m working for another company, we have the same need:<p>- Orchestrate hundreds of jobs<p>- Some need to start as soon as their dependencies are ready<p>- Others need to start according to timezone-based (and possibly dependencies)<p>- Notification on failures<p>- Make jobs fail if their dependencies are not ok after a given time<p>- Force a job status (i.e. set it &quot;ok&quot; even if it failed)<p>- Reasonable access control<p>- And if possible a nice dashboard!<p>While these requirements seem standard to me, I cannot find any acceptable solution:<p>- Cron is cool for the simplest things, nothing more<p>- Rundeck is nice for simple workflows, but not for complex dependencies<p>- I&#x27;m tinkering with SOS-Berlin Job Scheduler for days now, and I cannot find proper ways to orchestrate my dependencies<p>- Control-M looks over priced and certainly not deployable without months of project planing and contract negotiation<p>- Luidgi and Airflow are too much focused on data workflows, not on generic job scheduling<p>What are you using? :)
======
usernam3
For php laravel projects are using this service:

\- [https://github.com/cronboy/cronboy](https://github.com/cronboy/cronboy)

\- [https://cronboy.com/](https://cronboy.com/)

It doesn't fit needs of @pmchorus company, but is useful for simpler cases
when you need something more efficient than cron and more accurate than queue
delayed jobs. (Example: we use it for sending triggered email deliveries,
where batch processing of letters using cron is not possible, because of
amount of letters and cron schedule limit)

The idea is that you schedule job on specified time and cronboy invokes it via
http and execute it on your server.

Features:

\- Seconds accuracy of execution

\- Statistics on invoked jobs

\- Secure execution of your code

\- Retrying if job failed

\- Laravel friendly facade api

------
bigzen
Airflow might be primarily intended for workflows, but scheduling is fairly
robust. But you also say that you want more than just simple Cron so I suggest
taking another look.

What I really appreciate about Airflow is that it is self documenting. Anyone
can logon and see the exact workflow, schedule of that workflow, the status of
last few job runs and the python code that ran if they desire. Airflow
documentation itself is good and getting better.

It can run a simple cron job but it also gives you room to build it out into
much more. We use our Airflow scheduler to call simple API endpoints and then
that same server does daily loads into our data warehouse.

Also 1.8 just dropped to stable and it is a huge improvement upon earlier
versions.

------
benlovell
Sidekiq. It's a wonderful project and has formed the basis of so many projects
I have worked with over the years ever since it's inception. It's also a great
example of an OSS project with a healthy commercial offering.

------
Xeago
Given that you mention Luigi, have a look at Styx. Currently in use at Spotify
as a complement to the underlying job processing frameworks.

[https://github.com/spotify/styx](https://github.com/spotify/styx)

~~~
pmchorus
Looks like it only support cron expressions and no timezones. Moreoever, I
don't plan to deploy a Kubernetes cluster in the short term (looks like to be
a huge work to have something ready for production). I'm primarily looking for
something I can install on the existing systems.

------
jacksonnic
[https://www.nomadproject.io](https://www.nomadproject.io) is awesome for
scheduling both long running processes and time triggered jobs

~~~
pmchorus
It does not seem to handle dependencies.

------
pmchorus
Any experience with [http://www.schedulix.org/en](http://www.schedulix.org/en)
?

------
paktek123
[https://mesos.github.io/chronos/](https://mesos.github.io/chronos/)

~~~
pmchorus
I have no Mesos cluster running for now. I'm looking for something I can use
on the existing infrastructure. From what I read, it does not support
timezones.

------
aprdm
Celery might be what you looking for.

------
batbomb
look at slurm and other things based on it. Slurm plus mesos + airflow will
meet all your needs.

------
anilgulecha
Consider AWS lambda. It's not a monolithic system, but may work depending on
your use-case.

