Apache Iceberg Table Format

Apache Iceberg is an open table format for huge analytic datasets, designed for use with petabyte-scale data lakes. Originally developed at Netflix in 2017 by Ryan Blue and Daniel Weeks to solve data warehouse reliability and performance problems at massive scale, Iceberg became an Apache top-level project in 2020. Key features: schema evolution allowing adding, dropping, renaming, and reordering columns without rewriting data files, with full backward and forward compatibility for readers and writers. Hidden partitioning where partitioning is defined by table schema rather than data values, eliminating the need for users to maintain partition columns and reducing partition-related bugs. Partition evolution allowing changing partition layout over time without rewriting existing data, with the query engine handling multiple partition layouts transparently. Time travel enabling querying historical table states at any point in time via snapshot isolation, useful for auditing, debugging, and reproducible analysis. ACID transactions via snapshot-based isolation ensuring consistent reads even during concurrent writes, with optimistic concurrency control for multi-writer scenarios. Incremental processing for reading only new or changed data between snapshots, enabling efficient CDC (Change Data Capture) patterns. Metadata management with manifest files and manifest lists for efficient file pruning without scanning all data files. Catalog abstraction supporting Hive Metastore, AWS Glue, Nessie, REST catalogs, and custom implementations. Engine compatibility with Trino, Presto, Spark, Flink, Hive, and Impala for unified access. File formats including Parquet, ORC, and Avro. Maintenance procedures including compaction, expire snapshots, and remove orphan files. Used by Netflix, Apple, LinkedIn, and Stripe.