Show HN: Dcron – Distributed, fault tolerant job scheduling system

[victorcoder]

http://dcron.io/

[dozzie]

You may want to rename your project. dcron is Dillon's cron for quite a lot time already.

[feld → dozzie]

Yeah upon reading the title I thought Dillon had done some improvements on dcron.

[victorcoder → dozzie]

The project has been renamed to Dkron due to respect to Dillon's cron and to avoid confusion.

[jedisct1 → dozzie]

Yup, thought the same thing.

[victorcoder → dozzie]

Right I know, but if he don't mind, both projects are enough different to not create confusion.

Need more time to see if this will be necessary.

[mattzito → victorcoder]

Honestly, it's just the classy thing to do - the impact of changing your name is nominal, the potential confusion with the other Dcron is significant.

[dozzie → victorcoder]

No, they are not different enough. Both are cron daemons. Both are in the end supposed to be packaged. It's really enough to make the newer one (newer by decades!) to change its name.

[Abundnce10]

I have a bunch of Python scripts that run a various intervals (hourly, daily, monthly, etc). I have trouble keeping track of which jobs fail -- unless I notice an email doesn't show up, or a metric is missing. Is Dcron something I would use to keep track of my cron jobs and whether or not they ran successfully?

At this point I don't need my jobs to be distributed, I'm just looking for a way to keep track of them all and visualize characteristics of the job (start time, length, successful, etc). Ideally, I'd have a way to re-run the job within the UI.

[fortyfivan → Abundnce10]

Disclosure: I work for Iron.io

If you're open to a hosted solution, check out IronWorker: http://www.iron.io/worker

It's an async task processing service with a built-in job scheduler. You can upload your python scripts to Iron.io, then set schedules and other triggers to execute on-demand. We have a dashboard to manage tasks and schedules, see what ran and what failed, and you can visualize the characteristics you're looking for. We do distribute the workloads for you, but sounds like it could be a good fit.

[abrookewood → Abundnce10]

You can also try cronitor and others like it. You add a CURL command at the end of your script that 'touches' a HTTPS endpoint. If your script doesn't check-in at pre-defined intervals, you get alerted. They are perfect for situations where you don't actually want to set up any infrastructure.

[JonM → Abundnce10]

Hey,

I've been tinkering with this concept recently, it's not quite ready for beta testing, but I'd be happy to take feedback on what I've got so far: http://croncloud.io/

[victorcoder → Abundnce10]

Sure, it can do this well. It can run in a single dcron/etcd node if you don't need it to be fault tolerant. At this point you should be checking the job status in the ui as it doesn't have notifications yet.

[yashap → Abundnce10]

Check out Airflow (open source project from airbnb that does all of that very well).

[endymi0n]

Man, how I wish Kala and Dcron would just work together.

https://github.com/ajvb/kala

https://github.com/victorcoder/dcron

Fusing both would be AWESOME (I'd even like to help).

What Dcron gets right:

- Distributed by design

- Raft for the reliable parts, gossip for lightweight sync, a match made in heaven

- Linux mentality

- Dashboard included

- Good docs, nice website, looks serious

What Kala gets right:

- Built much more directly after Chronos, which has an amazing and battle tested API

- Using ISO durations instead of NIHing their own format

- Dependent jobs

[schmichael → endymi0n]

While Metafora doesn't support cron-like syntax for job scheduling or tag-based node targeting (yet!), it does support distributed fault tolerant task running. We hope to build out a webui soon as well.

However since Metafora is written as a Go library and provides a statemachine implementation for task handlers to use, it requires writing your distributed tasks in Go.

https://github.com/lytics/metafora

[nemothekid]

I haven't done any digging, but I'm assuming this uses etcd for membership? How does that work - how does a server know whether its a leader, and how does failover work?

[victorcoder → nemothekid]

Nope, it uses serf for membership.

[nodesocket]

How do I define jobs? Can they be written in any language? I didn't see anything in the documentation.

[victorcoder → nodesocket]

Added to the docs, big omission, sorry! http://dcron.io/docs/api/#create-or-update-a-job

[nodesocket → victorcoder]

Any change that dcron could manage the job scripts itself? I.E. you store the raw command scripts in dcon, then reference them in the create a job endpoint?

For example:

    POST /scripts --data 'node /home/node/myJob.js'
        

That returns a script id { "script_id": "507f1f77bcf86cd799439011" }

Then:

    POST /jobs --data '{
       "name":"cron_job",
       "schedule":"@every 2s",
       "script": "507f1f77bcf86cd799439011",
       "owner":"foo",
       "owner_email":"foo@bar.com"
    }'

This way you don't have to manage/create the scripts on each node manually.

[victorcoder → nodesocket]

I think you missed the point, jobs are stored in etcd so they're distributed and all dcron (server) nodes will see it. You can even create jobs from any node. Note that, non server nodes doesn't need access to etcd.

[andyeskridge → nodesocket]

I was looking into that too. Based on this job.json[0] definition from the git repo, it looks like it just executes the command. So I believe you could have the job written in any language.

But I think this also means you have to copy every job executable to every server that might have a chance of running the job.

[0] https://github.com/victorcoder/dcron/blob/master/job.json

[kylequest → nodesocket]

Looks like it's defined using the format you see in this json file: https://github.com/victorcoder/dcron/blob/master/job.json

The "command" field is the most important part. What you put there gets executed like this by their agent (on Linux): /bin/sh -c <your_scheduled_command_name>

[carl_]

Cool! Might use this when I renovate http.sh