Hacker News new | past | comments | ask | show | jobs | submit login

Check out Luigi (Python -- https://github.com/spotify/luigi).

I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.




There was a good tutorial at PyCon this past weekend called Workflow Engines Up and Running [1] on Python workflow automation frameworks, specifically comparing Luigi vs Airflow. The video is on YouTube as well [2].

[1]: https://us.pycon.org/2018/schedule/presentation/58/

[2]: https://youtu.be/kw0RL9LZk9s


Also using luigi here in production since 2015. We use it to manage a multi-day pipeline that is essentially semi-automatic (requires occasional human intervention).




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: