
Show HN: Clockwork- Distributed, Scalable Job Scheduler - uttpal
https://cynic.dev/posts/clockwork-scalable-job-scheduler/
======
palansh
Great post! In the last part of your blog you've mentioned: "To scale out we
can increase number of partition in kafka topic and add more clockwork nodes."

Above has a limit. In general, more partitions in a Kafka cluster leads to
higher throughput. However, one does have to be aware of the potential impact
of having too many partitions in total or per broker on things like
availability and latency.

How do we handle too many kafka partitions?

~~~
uttpal
Adding more partitions will definitely have impact on latency and
availability, and you will always have a trade-off between throughput and
latency however, Clockwork will work fine for most of use cases as kafka
supports upto 200k partitions per cluster
([https://blogs.apache.org/kafka/entry/apache-kafka-
supports-m...](https://blogs.apache.org/kafka/entry/apache-kafka-supports-
more-partitions)). Beyond that we will need to have multi-cluster support,
definitely in the pipeline __wink __

------
uttpal
Clockwork-Scheduler is a general-purpose distributed job scheduler. It offers
you horizontally scalable scheduler with at least once delivery guarantees. It
is fault-tolerant, persistent, easy to deploy and maintain.

Thanks for checking, would love to discuss

~~~
bogomipz
Thanks for posting, Loved the graphics and layout. They made for enjoyable
reading. However maybe you could add or share the motivation for creating the
project? Were their shortcomings of existing job scheduler solutions that you
evaluated? Is there an edge case that you have that existing solutions didn't
address etc?

~~~
drathi5
One of the popular existing job scheduler is Quartz, it is solid and does work
pretty well up to a limit. The problem arises where we need to scale-out from
a single master DB, there are some workarounds for that, but each has its
limitations. ([https://hazelcast.com/blog/distributed-task-coordination-
wit...](https://hazelcast.com/blog/distributed-task-coordination-with-
hazelcast/))

While there are some existing solutions like BigBen from Walmart, it might be
possible that they were overkill for the task required.

