File Formats · Open Table Formats

Lance

Assess Community Lance community (independent open source, LF-inspired governance) Since 2022

Judgement: Assess

AI/ML-optimised columnar format; trajectory good, adoption concentrated.

Modern columnar file format optimised for AI/ML workloads: 100x faster random access than Parquet, native support for embeddings and multimodal data (images, video, audio), and zero-copy schema evolution without rewriting files.

Pairs with the Lance table format and catalog spec to form a lakehouse stack purpose-built for vector search and feature engineering alongside SQL analytics.

Why it counts as a standard

Lance's on-disk format is published as an open spec with independent Rust and Python implementations. LanceDB, DataFusion, DuckDB integrations, and downstream tools target the format itself rather than a single library — putting it on the de-facto standard track under an open, LF-style community governance model.

Example

A columnar format built for random access: write once, then point-query rows without a full scan.

import lance
import pyarrow as pa

tbl = pa.table({
    "id": [1, 2],
    "embedding": [[0.1, 0.2], [0.3, 0.4]],
})
lance.write_dataset(tbl, "/tmp/embeddings.lance")

dataset = lance.dataset("/tmp/embeddings.lance")

At a glance

Category
File Formats, Open Table Formats
Governance
Lance community (independent open source, LF-inspired governance)
Status
Emerging; production use concentrated in AI/ML stacks
First released
2022

Links

Related standards

Other standards in File Formats and Open Table Formats.

See Lance in context

Open the interactive Data Landscape to compare Lance against every other open standard, or grab the raw JSON. Spotted something wrong? Open an issue.