
Introducing Chronos: A Replacement for Cron - AirbnbNerds
http://nerds.airbnb.com/introducing-chronos
======
tptacek
I'm surprised there aren't more things like this, because Cron replacements
are really valuable.

We wrote a small C program that serves as a scheduling daemon with Redis; we
have a keyspace in Redis that can be used to schedule millisecond-granular
periodic or one-shot tasks with a flexible "Chronic"-like specification
syntax.

It is hugely more convenient and usable than cron. Once you have it, you
immediately spot lots of opportunities to factor systems into scheduled jobs
that you might have avoided doing if it meant you had to deal with cron.

~~~
minimax
I don't think it's uncommon at all to roll your own job scheduler. I know I've
done it at the last two places I have worked, though neither had nearly as
fancy a front-end as Chronos appears to have. This particular scheduler
(Chronos) looks like a cross between traditional cron and HPC cluster job
scheduling systems (a la Sun Grid Engine). It looks like a really cool
project.

~~~
tptacek
That depends on what you mean by job scheduling, right? Lots of people would
call Resque a job scheduling system, but they still reach for cron when they
want to run something every 5 minutes.

~~~
minimax
I'm about as unaware of the Ruby ecosystem as anyone who regularly reads HN
can be, so while I've heard of Resque I'm not really sure where it fits in.

I suppose the definition of job scheduling is dependent on what your workload
entails. Is it something you expect to complete in less than a second, a
minute, or a big resource intensive HPC job you expect to run for hours? What
sorts of dependencies do you have between the things you want to run? Will it
run in the main event loop, separate thread, separate process, or on a
separate host? I think this will reflect in the tool you choose for
scheduling, or cause you to write your own.

It sounds to me like the target workload for Chronos is big slow batch jobs
(ETL, data analysis). HPC job schedulers take care of things like resource
management (run this job on this machine because it has the lowest load) and
dependency management (don't run job b until job a completes), whereas cron
handles running periodic tasks and that's about it. Chronos is nice because it
looks like it is combining the two and includes a nice looking web based
control panel to boot.

------
zobzu
so... cron's a 200kb self contained C program, super simple, does one thing,
and does it well.

Chronos is built on top of frameworks, needs a few services to run.

Certainly it has value, but marketing it as a cron replacement is wrong imo.
Chronos is one of these "web-server-service-tool" thingie, but no, it's not
cron.

~~~
bitwize
Every time I come across one of these heavyweight, dependency-encrusted
"replacements" for basic Unix tools, I'm reminded of L33tStart[0], a
fictional, satirical init(1) replacement I wrote about on Reddit -- a sort of
parody of systemd.

Every day, in a bizarre manifestation of Poe's Law, L33tStart seems less and
less a parody.

[0]
[http://www.reddit.com/r/programming/comments/14ay0r/hacker_k...](http://www.reddit.com/r/programming/comments/14ay0r/hacker_koan_on_the_proliferation_of_unix_text/c7bntjn)

~~~
adambenayoun
Any chance you could open source L33tStart. Also what type of ponies were you
using for your startups? Black or white ponies?

~~~
andrewflnr
Mostly pink and purple, is my impression.

------
eksith
This is almost like saying "A Replacement for Gravity".

The benefit having something so simple and singular in function with no
dependencies is that there's so little to break. While I appreciate the need
for having something with more capability, calling it a "replacement" is a bit
facetious.

I wish them success and call it an airplane. I'll stick to my hang glider.

~~~
dice
What the ops team isn't telling them (DevOps only goes so far) is that there's
a cron job on each node which periodically bounces the Chronos processes (and
its 50 dependencies) in order to mitigate memory leaks.

------
hendzen
Interesting to see that this was built on top of Apache Mesos, a dynamic
cluster partitioning framework started by some researchers at Berkeley [1]. So
far I've heard about Twitter (and now Airbnb) using Mesos in production. Is
anyone else evaluating or using it currently?

[1] - <http://incubator.apache.org/mesos/papers/nsdi_mesos.pdf>

~~~
_harry
Wired had a great article on Mesos that mentions Conviva also uses Mesos:
[http://www.wired.com/wiredenterprise/2013/03/google-borg-
twi...](http://www.wired.com/wiredenterprise/2013/03/google-borg-twitter-
mesos/all/)

------
stfp
Looks great, but calling it a replacement for cron is like calling word a
replacement for cat.

------
verelo
This is brilliant! I've had to do a lot of work creating queue based systems
in recent years, and i really like off the shelf solutions to problems like
this. Cron is a great tool, but its a mess once you have a bunch of machines
and it really just does not scale.

~~~
bdunbar
By itself it does not. If you have a bunch of machines, you also have a config
system (right?) and they (puppet, chef and so on) have the means to manage
cron tasks as a class.

You still get cron, and the means to manage them across one's servers.

------
stiff
I really hope someone will succeed in rolling out a good and eventually
popular replacement for Cron, because Cron completely sucks.

The crontab file has a weird syntax, that in some respects seems to look like
a shell script, but isn't really one and some shell constructs work, other
not. There are lots of ways you can make a mistake in writing the commands to
be executed in a way that the command you intended won't run but will fail
silently and you won't get any trace of an error happening. It is hard to even
extract some common parts of commands and put them into a variable. I wasted
hours and hours debugging weird cron errors. One case was where the crontab of
one of the users was moved over to be the system crontab and strangely didn't
work. Well, turns out the system wide crontab has one more field, but cron
will not signal an error even in an obvious case like this, it will just fail
silently (and this is the thing cron is really good at in my experience).

------
DannoHung
Awesome! If anyone's ever had to use Ctrl-M or AutoSys, you know well the
supreme horridness of other solutions in this space.

~~~
newman314
Brr, AutoSys gives me nightmares.

------
philp
I know a couple guys that will be rather unhappy about the choice of moniker.
<http://www.getchronos.com/>

~~~
signed0
<http://chronoss.sourceforge.net/> has been around longer than they have.

~~~
lsiclait
<http://en.wikipedia.org/wiki/Chronos>

~~~
mbubb
Love it. The god Chronos has been around longer than time itself...

------
solidsnack9000
Tools like this can make a lot of ops housekeeping stuff easier, like log roll
ups and deleting old backups and so forth. One doesn't want to depend on
particular machines being up; but on the other hand, there's not a 99.999999%
uptime requirement.

------
3amOpsGuy
Always happy to see more tools like this - the bigger the toolbox, the easier
our lives are.

Of the various approaches I've had to depend on in recent memory, from perl
scripts querying a central Db right through to eye wateringly expensive
Control-M or Autosys in larger envs, It's plain old cron, fronted by config
management (cfengine, puppet, chef, salt - it doesn't matter which) that has
proven most dependable, easiest to train others on and simplest to debug.

------
erikbern
We've built Luigi at Spotify to solve a lot of similar problems:
<https://github.com/spotify/luigi>

Might be worth checking out if you are building large data flows. We probably
run 10k Luigi "tasks" every day, of which the majority is Hadoop jobs. They
are all organized in a large dependency graph expressed within Python, and you
also get visualization, exception handling, atomic file operation, etc.

------
Zenst
Many people do much of the fancy stuff with system managment products (Tivoli
or BMC patrol spring to mind from past experiences). With that they can do
centralised monitoring of the server and services and software and automate
responses to defined conditions allowing automatic responses with the right
thought and alerting callout when exceptions to the rules occur and when
things need looking at.

Now all that said, I'm not uptodate on that side of things and even less
uptodate on open source alternative, though my quick look at this does
indicate that it is a start in the right directon and can only get better. So
quick look and glance over gave me a good gut feeling, which is always nice to
have.

------
revscat
Anyone know why launchd hasn't taken off in this regard? I've used it on my
personal systems with success, but rarely (if ever) see it mentioned as a cron
replacement. Instead, you frequently see these solutions done from scratch.

~~~
andrewflnr
Link for the ignorant and lazy, as I once was:
<http://en.wikipedia.org/wiki/Launchd>

Seems like a decent idea. According to Wikipedia, Ubuntu considered it in 2006
when they were looking for a new unit system, but didn't like that it was
under Apple's own license, which was shortly thereafter changed to the Apache
license.

I'll remember it if I want to do something unconventional on a system.

------
rjurney
I've seen it in action, and it can schedule down to the milisecond across
machines.

------
nemesisj
And....that blog is going offline in a few weeks. Thanks Posterous/Twitter!

~~~
benatkin
pls...they used a custom domain and they have plenty of time to migrate to a
new blogging service and point their DNS to it

------
sandGorgon
Im not sure what mesos brings to the table (possibly distributed dependency
management), but for a powerful replacement for cron with dependency
management, logging, etc. one could use systemd timers [1]

1\. [http://jason.the-graham.com/2013/03/06/how-to-use-systemd-
ti...](http://jason.the-graham.com/2013/03/06/how-to-use-systemd-timers/)

------
philipcristiano
I was planning to do something similar after I learned about Mesos although I
haven't found the time. At the moment I'm using a few processes with MySQL
publishing with RabbitMQ to handle distribution and node failures. It doesn't
have nearly as nice an interface but RabbitMQ isn't terribly difficult to
manage. Thanks for open sourcing!

------
darose
Seems like Chronos covers a lot of the same functionality as Rundeck.

------
tszming
I am using a similar tool for job automation, history & alert, and it is
called Jenkins :)

~~~
gzur
Yeah, I'm amazed that people consistently run around reinventing the wheel,
when there are 18-wheeled chariots out there free for the grabbing.

------
gnuvince
As far as Cron replacements go, I prefer whenjobs [1].

[1] <http://people.redhat.com/~rjones/whenjobs/>

------
dschiptsov
in Java?

Something is deeply wrong with this world.)

