Apache Beam

Apache Beam is an open-source unified programming model for defining and executing data processing pipelines, originally developed at Google as the Dataflow SDK and donated to the Apache Software Foundation in 2016. The name Beam comes from the combination of Batch and strEAM processing, reflecting its core design goal of providing a single API that works for both batch and streaming data processing use cases. Apache Beam's key innovation is its abstraction of the data processing pipeline from the execution engine (called a Runner), allowing developers to write pipeline code once and run it on multiple distributed processing backends without modification. Supported Runners include Apache Flink, Apache Spark, Google Cloud Dataflow, Amazon Kinesis Data Analytics, Hazelcast Jet, and the Direct Runner (for local testing). The Beam programming model is built around several core concepts: Pipeline (the logical graph of processing operations), PCollection (a distributed dataset representing data flowing through the pipeline, which can be bounded for batch or unbounded for streaming), PTransform (operations that transform input PCollections into output PCollections, including ParDo for parallel processing, GroupByKey for aggregation, and windowing), Window (defining how unbounded data is divided into finite chunks based on time or other criteria), and Triggers (controlling when windowed data is emitted). Beam supports advanced streaming features like event-time processing, watermark tracking, allowed lateness, and session windows. Available SDKs include Java, Python, Go, and SQL. As of 2026, Apache Beam is used at massive scale by companies like Google, Spotify, and PayPal.