Apache Hudi Data Lake Platform

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lake platform that brings ACID transactions, record-level upserts and deletes, and change streams to big data workloads. Originally developed at Uber in 2016 by Vinoth Chandar to solve real-time analytics and data pipeline challenges, Hudi became an Apache top-level project in 2020. Key features: two table types including Copy On Write (COW) where updates rewrite affected parquet files on every write operation for maximum read performance, and Merge On Read (MOR) where updates are written to row-based log files and merged during reads or compaction for maximum write performance. Record-level upserts and deletes enabling efficient CDC (Change Data Capture) ingestion from databases into the data lake, with primary key-based deduplication and fast point lookups. ACID transactions ensuring atomicity and consistency for concurrent readers and writers with snapshot isolation. Incremental queries (pull) for reading only data that changed since a specific timestamp, enabling efficient streaming patterns without re-scanning the entire table. Time travel queries for accessing historical data states at any timestamp or instant. Schema evolution and compatibility for safely adding, renaming, and dropping columns. Clustering for optimizing data layout by co-locating frequently co-accessed records, improving query performance. Compaction for merging log files (MOR tables) into columnar base files in the background. Table services including clean (removing old file versions), archive (moving old metadata to archive), and rollback (aborting failed writes). Multi-table transactions for atomically writing to multiple tables. Hive Metastore, Glue Catalog, and custom catalog integration. Engine support for Spark, Flink, Java, and Python (via PyHudi). Used by Uber, ByteDance, Walmart, and EMIS Health.