Transformation — how data is processed and reshaped

In-Memory Format

2 open standards for In-Memory Format in a modern data architecture, each with an opinionated judgement: Adopt, Situational, Assess, or Caution.

Start with DataFrame API and Apache Arrow.

Adopt 2 standards

The standard to reach for in new work. Proven, multi-vendor, clearly the default for its slot.

DataFrame API — Python Dataframe API Standard
Portable Python DataFrame spec; the way to write library code that runs on pandas, Polars, Modin, and cuDF.

Consortium for Python Data API Standards (data-apis.org)
Apache Arrow
In-memory columnar standard powering Flight, ADBC, DataFusion, Polars.

Apache Software Foundation

More in Transformation

Transformation covers how data is processed and reshaped.

Processing

See In-Memory Format in context

These standards are one panel of the interactive Data Landscape, which maps every open standard a modern data architecture is built on. The underlying data is a single JSON file; disagree with a judgement? Open an issue.