Spark
Apache Spark
Judgement: Adopt
Default distributed batch+streaming engine.
Distributed analytics engine with a DataFrame API across Scala, Java, Python, and R. The classic large-scale processing engine for batch and streaming workloads.
Spark Connect decouples client and engine, making the DataFrame API a portable client interface — which is why it shows up here as an interface, not just an engine.
Why it counts as a standard
Spark's DataFrame API and the Spark Connect protocol are the standard parts. Spark Connect publishes a gRPC contract that any Spark-compatible engine can serve, and the DataFrame API is what other engines (e.g. Databricks Connect, Apache Sedona) target. The interface, not the JVM runtime, is what makes Spark a standard.
At a glance
- Category
- Processing
- Governance
- Apache Software Foundation
- Status
- Stable; Spark 4.x current, 3.x widely deployed
- First released
- 2014
Links
Related standards
Other standards in Processing.
- dbt — data build tool
- Pandas — pandas
- SQL DML — SQL Data Manipulation Language
- Beam — Apache Beam
- Ibis
- XSLT — Extensible Stylesheet Language Transformations
See Spark in context
Open the interactive Data Landscape to compare Spark against every other open standard, or grab the raw JSON. Spotted something wrong? Open an issue.