Delta Lake Storage Framework

Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads, enabling reliable data lakes at scale. Originally developed at Databricks (founded by the creators of Apache Spark) and open-sourced in 2019, Delta Lake has become one of the most popular data lake table formats alongside Apache Iceberg and Apache Hudi. Key features: ACID transactions ensuring data integrity for concurrent reads and writes, with serializable isolation level preventing dirty reads, non-repeatable reads, and phantom reads. Schema enforcement and evolution preventing bad data from entering the table while allowing safe schema changes (add columns) without rewriting existing data. Time travel enabling querying previous versions of data via version numbers or timestamps for audit, debugging, and reproducible ML experiments. Unified batch and streaming APIs where the same Delta table can be read and written by both batch (Spark batch) and streaming (Spark Structured Streaming) workloads. Upserts, deletes, and merges supporting MERGE INTO, UPDATE, and DELETE operations for data governance, GDPR compliance, and slowly-changing dimensions. Z-ordering and data skipping for optimizing query performance by co-locating related data and skipping irrelevant files based on statistics. Auto-compaction for automatically merging small files during writes. Optimized writes for coalescing data before writing. Checkpointing for efficient state recovery in streaming workloads. Change Data Feed for capturing row-level changes. Unity Catalog integration for fine-grained access control. Delta Standalone (delta-standalone) for reading Delta tables from non-Spark Java applications. Delta Rust API for native Rust integration. Connector ecosystem with Presto, Trino, Flink, Snowflake, BigQuery, Athena, and Redshift.