I've had issues with deadlocks in airflow as well -- and it's hard to know this is happening without looking at the scheduler log. I eventually hooked up airflow's logging to rollbar so I'd know this was going on at all.
I love the telemetry you get in Airflow's ui, but I don't like the complexity of the scheduler and database-backed backend. Luigi is superior here imho using just a filesystem for managing state.
I love the telemetry you get in Airflow's ui, but I don't like the complexity of the scheduler and database-backed backend. Luigi is superior here imho using just a filesystem for managing state.