Temporal has really solved so many problems for us it is only opinionated about a few things that actually matter, and gives you complete flexibility otherwise.
The days of Airflow and similar seem like a stone age in comparison.
Major benefit of Airflow is the number of already implemented integrations. Importing data from GCS to BigQuery, copying data from Postgres to GCS, KubernetesPodOperator and so on. IIUC with Temporal you get only workflow management which can be easily integrated with any application to implement business logic. And this is great, because implementing business workflow in Airflow is even more awful than the Airflow itself.
But for any ETL or plumbing job Airflow is IMO better due to existing integrations.
You are correct, that's the main difference. I wrote some more on the topic of
data workflow engines like Airflow vs general-purpose application development workflow engines like Temporal here: https://community.temporal.io/t/what-are-the-pros-and-cons-o...
Is Temporal meant to be an Airflow replacement? The website exclusively offers examples of executing multi-step core business logic and not ETL workflows.
It’s for whatever your code can do, same as Airflow.
One of the great things a Temporal workflow can do is can send or wait around for signals from external processes, indefinitely if needed. It’s much easier to start orchestrating things you already have. You don’t really need to buy into it as much as you do with Airflow. If it exceeds retries or timeouts, it can send a signal or launch a process to notify a human that something needs fixed, then someone can intervene, then notify the workflow that it can keep going now. Airflow is much more all-or-nothing success or failure in my experience. Very hard to re-enter the workflow after something got twisted.
Certainly Airflow has more ETL integrations at this point in part due to how much longer it’s been around and the use cases it’s been evangelized for.
I had never worked in place that had much investment in the ETL integrations, we used dockerized processes and just the docker operator, as it was easier to develop and test independant of an airflow instance.
So, when looking at Airflow some time ago it looked like it was good at 'fixed workflows', something like, fetch last day's data from a website, process then load it at the DB
But it seemed it was bad at more flexible ones, like, load data, then process each entry in a certain way, then trigger a new workflow based on each entry (like send an email to every entry on the data fetched based on some condition)
The other useful pattern is always running workflows that can be used to model lifecycle of various entities. For example you can have an always running workflow per customer which would manage its service subscription and other customer related features.
The days of Airflow and similar seem like a stone age in comparison.