Distributed Pub Sub Messaging

A distributed, open source pub-sub messaging and streaming platform designed for high-performance, geo-replicated streaming across multiple data centers, currently at version 5.0.0-M1 with over 14,800 GitHub stars, 740-plus contributors, and adoption by organizations serving billions of messages per day. The platform delivers millions of messages per second with end-to-end latency consistently below 10 milliseconds, achieved through a layered architecture that separates stateless brokers from stateful storage nodes called BookKeepers, enabling independent scaling of compute and storage. The compute-storage separation also enables fast broker failover, as new brokers take over topic ownership without data migration, with failover times measured in seconds rather than minutes. Native geo-replication synchronizes data across geographically distributed clusters with configurable consistency guarantees, supporting active-active deployments where producers and consumers operate simultaneously in multiple regions. Multi-tenancy is built into the core architecture, with each tenant having isolated topic namespaces, authentication, authorization, and rate limiting, eliminating the need for separate broker instances per tenant. The flexible messaging model supports both queuing through exclusive, shared, failover, and key-shared subscription types, and streaming through durable cursor-based message positioning. Tiered storage automatically moves older messages from hot BookKeeper storage to cold object storage like Amazon S3 or Google Cloud Storage, reducing costs for long-term data retention while maintaining transparent access. Functions computing enables lightweight stream processing with Python and Java functions deployed directly on the broker or in separate workers. The platform integrates with Kafka through the KoP protocol compatibility layer, enabling migration without client code changes.