Dagster

Dagster is a data orchestration platform designed to help teams build, schedule, and monitor reliable data pipelines. It provides a unified framework for managing the entire lifecycle of data assets, from development through production, with a focus on modularity, observability, and testability. At its core, Dagster allows users to define data pipelines as collections of interconnected assets, where each asset represents a dataset, a table, a machine learning model, or any other data product. Instead of writing traditional DAG (directed acyclic graph) definitions with separate tasks and dependencies, users describe the relationships between assets directly. This asset-centric approach makes it easier to understand lineage, track dependencies, and ensure data freshness. The platform includes a built-in scheduler that supports cron expressions, time-based triggers, and sensor-based triggers (e.g., waiting for a file to land in S3 or a new row in a database). Users can define schedule intervals or use event-driven sensors to kick off pipelines automatically. Dagster also handles backfills and partitioned runs, enabling efficient processing of large time-series data. For monitoring and debugging, Dagster provides a rich web UI called Dagit. The UI displays the full asset graph, run history, logs, and asset materialization status. Each asset can have metadata such as its partition mapping, upstream/downstream dependencies, and custom tags. Users can inspect individual runs, view error traces, and visualize data quality checks in real time. Data quality is a first-class feature: users can attach expectations (e.g., “column x should not contain nulls” or “row count should be between 1,000 and 10,000”) directly within asset definitions. These expectations are evaluated during p