Transformation — how data is processed and reshaped

Processing

7 open standards for Processing in a modern data architecture, each with an opinionated judgement: Adopt, Situational, Assess, or Caution.

Start with dbt, Spark, Pandas and SQL DML.

Adopt 4 standards

The standard to reach for in new work. Proven, multi-vendor, clearly the default for its slot.

  • dbt — data build tool

    Analytics-engineering default: SQL-first models, tests, lineage.

    dbt Labs (vendor-driven open source)

  • Spark — Apache Spark

    Default distributed batch+streaming engine.

    Apache Software Foundation

  • Pandas — pandas

    Universal Python DataFrame; you don't choose it, you encounter it.

    NumFOCUS (open source)

  • SQL DML — SQL Data Manipulation Language

    Portable transformation language across relational/lakehouse engines.

    ANSI / ISO/IEC 9075

Situational 1 standard

The right answer in some contexts but not others. Pick deliberately based on the constraint.

  • Beam — Apache Beam

    Write once, run on Flink/Spark/Dataflow; right when runner portability matters.

    Apache Software Foundation

Assess 1 standard

Promising but not yet proven for production-default use. Track it and prototype, but don't commit your architecture.

  • Ibis

    Portable Python DataFrame compiling to many backends; trajectory strong, footprint small.

    Ibis Project (open source)

Caution 1 standard

We'd avoid it for new work — superseded or fading, but still encountered in existing systems.

  • XSLT — Extensible Stylesheet Language Transformations

    XML transformation; maintain if you have it, don't pick it new.

    W3C

More in Transformation

Transformation covers how data is processed and reshaped.

See Processing in context

These standards are one panel of the interactive Data Landscape, which maps every open standard a modern data architecture is built on. The underlying data is a single JSON file; disagree with a judgement? Open an issue.