Storage — where data lives

File Formats

8 open standards for File Formats in a modern data architecture, each with an opinionated judgement: Adopt, Situational, Assess, or Caution.

Start with CSV, JSON, XML, YAML, PARQUET and AVRO.

Adopt 6 standards

The standard to reach for in new work. Proven, multi-vendor, clearly the default for its slot.

  • CSV — Comma-Separated Values (RFC 4180)

    Universal flat file; ugly but ubiquitous.

    IETF (RFC 4180)

  • JSON — JavaScript Object Notation

    Universal payload format; not a choice.

    IETF RFC 8259 / ECMA-404

  • XML — Extensible Markup Language

    Load-bearing in finance, healthcare, and government; mature tooling around XSD, XSLT, XPath, XQuery.

    W3C

  • YAML — YAML Ain't Markup Language

    Default surface for config and contract specs.

    yaml.org / community spec

  • PARQUET — Apache Parquet

    De-facto columnar storage; baseline for analytics.

    Apache Software Foundation

  • AVRO — Apache Avro

    Kafka's binary payload of choice; first-class with Schema Registry.

    Apache Software Foundation

Situational 1 standard

The right answer in some contexts but not others. Pick deliberately based on the constraint.

  • ORC — Apache ORC (Optimized Row Columnar)

    Pick if you're in Hive/Tez territory; otherwise Parquet wins.

    Apache Software Foundation

Assess 1 standard

Promising but not yet proven for production-default use. Track it and prototype, but don't commit your architecture.

  • Lance

    AI/ML-optimised columnar format; trajectory good, adoption concentrated.

    Lance community (independent open source, LF-inspired governance)

More in Storage

Storage covers where data lives.

See File Formats in context

These standards are one panel of the interactive Data Landscape, which maps every open standard a modern data architecture is built on. The underlying data is a single JSON file; disagree with a judgement? Open an issue.