On: Joining a billion rows 20x faster than Apache Spark

[makapuf]

Pardon my ignorance but what would you use Jenkins for ? Scheduling ?

[tetha → makapuf]

Jenkins in such a setting gives you two good things: i) scheduling. and ii) access control. The ability to give random dude X the ability to trigger computation Y, Z, and A, without the ability to change said computations.

[usgroup → makapuf]

Triggering jobs more generally (schedule or push notification) and splitting things into jobs and/or pipelines.