Beam
Apache Beam
Judgement: Situational
Write once, run on Flink/Spark/Dataflow; right when runner portability matters.
Unified programming model for batch and streaming data processing. Pipelines written once in Beam can run on Flink, Spark, Google Cloud Dataflow, or the direct local runner.
Originated as Google's Dataflow SDK, donated to ASF in 2016. Stronger claim to “standard” than most processing frameworks: the model itself — PCollections, transforms, windowing, triggers — is the spec, with runners as competing implementations.
Why it counts as a standard
Beam's value is the model, not the runner. The Beam programming model and the portability framework (Fn API, Runner API) are an open specification that multiple runners implement. Pipelines move between Flink, Spark, and Dataflow without code changes precisely because all three implement the same Beam contract — making the model the standard surface tools and teams target.
At a glance
- Category
- Processing
- Governance
- Apache Software Foundation
- Status
- Stable; multi-runner model spec
- First released
- 2016
Links
Related standards
Other standards in Processing.
- dbt — data build tool
- Pandas — pandas
- Spark — Apache Spark
- SQL DML — SQL Data Manipulation Language
- Ibis
- XSLT — Extensible Stylesheet Language Transformations
See Beam in context
Open the interactive Data Landscape to compare Beam against every other open standard, or grab the raw JSON. Spotted something wrong? Open an issue.