Apache Pulsar Messaging System

Apache Pulsar is a free and open-source distributed pub-sub messaging and streaming platform originally developed at Yahoo (by a team led by Joe Francis, Matteo Merli, and Jai Asher starting in 2012) to handle Yahoo's massive messaging needs across products like Mail, Finance, and Sports. Open-sourced in 2016, became an Apache Top-Level Project in 2018. Key features: architecture: separates compute (brokers) from storage (Apache BookKeeper). Brokers handle message routing and delivery; BookKeeper handles durable storage in replicated log streams (ledgers). This separation enables independent scaling. Topics: persistent, non-persistent, and partitioned topics. Partitioned topics distribute messages across brokers for higher throughput. Subscription modes: exclusive (one consumer), shared (round-robin), failover (primary/backup), and key_shared (consistent key-to-consumer for ordered processing). Multi-tenancy: native multi-tenancy with tenants, namespaces, and authentication at each level. Geo-replication: replicate topics across geographically distributed clusters for disaster recovery. Pulsar Functions: lightweight compute for stream processing with exactly-once semantics. Pulsar IO: connectors for importing from external systems (Kafka, Kinesis, Cassandra) and exporting to databases and queues. Pulsar SQL: query topics using Presto/Trino. Schema registry: built-in, supporting Avro, JSON, Protobuf with backward/forward compatibility. Tiered storage: offload cold data to S3, Azure Blob, GCS. Java. Apache-2.0.