Why Not Dagster

The Orchestrator Landscape

Airflow is the established standard with the biggest community. Dagster is the newer contender that rethinks how orchestration should work. Prefect is out there too, along with a few others. The question is not "which is objectively best" because that depends entirely on your team, your stack, and how much pain you are willing to absorb during a migration.

The real question: does switching solve a problem you actually have, or does it just feel like progress?

Assets vs Tasks: The Core Difference

This is where Dagster diverges from Airflow in a fundamental way. Airflow thinks in tasks: do this, then do that, in this order. Dagster thinks in assets: this table depends on that table, materialise them when the upstream data changes.

The asset model is genuinely a better mental model for data pipelines. You are describing what your data should look like, not what steps to run. Lineage comes for free because the dependencies are explicit.

But switching from task-based thinking to asset-based thinking is not trivial. If your team has been writing Airflow DAGs for years, it is a real mindset shift. And if you have hundreds of existing DAGs, you are not rewriting those overnight.

Development Experience

Dagster's local development story is better than Airflow's. You can run and test assets locally without spinning up a full scheduler. The type system catches errors before runtime. The UI is more modern and gives you better visibility into what is happening.

Testing is also different. Dagster was built with testability in mind from the start, which shows. Writing unit tests for Dagster assets feels more natural than testing Airflow operators.

Airflow has improved here with Airflow 2.x. But the gap is still real if you are comparing the day-to-day developer experience.

Community and Ecosystem

This is where Airflow wins, and it is not close. The community is massive. If you hit a problem, someone has probably already solved it and posted the answer on Stack Overflow. The operator ecosystem covers almost every service you can think of. Managed options (MWAA, Cloud Composer, Astronomer) give you multiple deployment paths.

Dagster's community is growing, but it is still smaller. Fewer blog posts, fewer conference talks, fewer battle-tested patterns from production deployments at scale. When you hit a weird edge case at 2am, community size matters.

Why I Didn't Switch

Not because Dagster is bad. It is not. The asset model is smart, the developer experience is better in several ways, and the architecture is more modern.

I did not switch because the cost of switching outweighed the benefits. Existing DAGs that work. A team trained on Airflow. Managed infrastructure already set up. Integration patterns already solved. The pain of Airflow is known and manageable. The pain of migrating is unpredictable and high.

If I were starting from scratch today with no existing codebase, would I consider Dagster? Honestly, yes. But "starting from scratch" is a luxury most teams do not have.