DataHub Metadata Platform

DataHub is an open-source metadata platform for data discovery, data observability, and data governance, designed to help organizations catalog their data assets, track data lineage, and manage data quality across complex data ecosystems. Originally developed at LinkedIn (by Shirshanka Das, Emmanuel Ifiong, and the Data Infrastructure team) starting in 2016 and open-sourced in 2020 under the DataHub Project, it is now managed by Acryl Data (founded by the original LinkedIn team, headquartered in Mountain View, California, and backed by $8M from Sofinnova Ventures and other investors). Key features: extensible metadata model (GMS - Generic Metadata Store) using a JSON-schema-based entity-relationship model that can describe any data asset (datasets, dashboards, charts, ML models, pipelines, metrics, containers, corpora) and their relationships (lineage, ownership, usage, deprecation). Ingestion framework: over 80 source connectors for metadata extraction from databases (MySQL, PostgreSQL, Snowflake, BigQuery, Redshift, ClickHouse, Delta Lake, Iceberg), streaming platforms (Kafka, Pulsar), BI tools (Tableau, Looker, Power BI, Metabase, Superset), orchestration platforms (Airflow, Dagster, Prefect, Spark), ML platforms (SageMaker, Vertex AI, Databricks), source control (dbt), and message schemas (Confluent Schema Registry, Avro, Protobuf). Real-time metadata streaming: push-based architecture using Apache Kafka for metadata change events. Lineage tracking: automatic data lineage via OpenLineage, Marquez, and Airflow integration. Search: Elasticsearch-based full-text search with faceted filtering. Data quality: integration with Great Expectations, dbt tests, and custom assertions. Access control: fine-grained role-based access control. GraphQL and REST APIs. React-based frontend. Deployable via Docker Compose, Helm, or managed Acryl Cloud. Apache-2.0.