Above has a limit. In general, more partitions in a Kafka cluster leads to higher throughput. However, one does have to be aware of the potential impact of having too many partitions in total or per broker on things like availability and latency.
How do we handle too many kafka partitions?
Thanks for checking, would love to discuss
While there are some existing solutions like BigBen from Walmart, it might be possible that they were overkill for the task required.
I will just point to major shortcomings here of quartz (the go to job scheduler) scaling beyond one node is difficult because
* as it uses db locks to adding more nodes does not scale linearly.
* we can add custom partitioning (consistent hashing) on top of multi node multi db setup, but maintaining such setup would be very difficult eg adding and removing nodes, handling node failure (will have to move data around from its db to let other node execute those schedules).