Dagster

Dagster is an AI-native DataOps platform for orchestrating, observing, and activating data pipelines across the entire data stack. Developed by Dagster Labs (founded by Nick Schrock, formerly of Facebook GraphQL team), Dagster introduces an asset-centric paradigm where data pipelines are defined in terms of the data assets they produce (tables, ML models, reports) rather than the tasks that execute them, providing automatic lineage tracking, dependency resolution, and freshness monitoring. The core abstraction is the Software-defined Asset (SDA), a Python function decorated with @asset that declares its dependencies on other assets, enabling automatic computation of the full DAG. Key features include: native dbt integration (syncing dbt models as assets, running dbt commands, and surfacing dbt test results as data quality checks), over 200 integrations (Snowflake, BigQuery, Redshift, Databricks, Spark, Pandas, Polars, Great Expectations, Fivetran, Airbyte, AWS, GCP, Azure), I/O managers for automatic materialization and caching, partition-based computation for incremental processing, backfilling, resource management (database connections, API clients with lifecycle hooks), schedules and sensors (event-driven triggers), code locations for multi-repo setups, branch deployments for safe CI testing, and a web UI (Dagster UI) for monitoring pipeline runs, asset lineage graphs, and data quality. Dagster+ Cloud offers managed deployment, hybrid execution, and Dagster+ AI for natural language pipeline diagnostics. As of 2026, version 1.9+ has over 12,000 stars.