Schema Evolution
Also known as: schema change, schema versioning.
Changing the structure of data over time while old data stays readable and existing consumers keep working.
Synthetic dataExamples on this page use synthetic data created for illustration.
Definition
Schema evolution is changing a dataset’s structure, by adding, renaming or removing fields, while existing data stays readable and existing consumers keep working. It depends on compatibility rules, such as allowing new optional fields but never silently changing what a field means.
In plain terms
Datasets live longer than their first design. New questions need new fields; old fields turn out to be wrong. The safe way to change is to know, for each kind of change, who breaks. Readers that ignore unknown fields survive additions; readers that depend on a removed or renamed field do not.
The most dangerous change is the one the schema cannot see: the same field name and type, with a new meaning. Nothing fails, and every number downstream is quietly wrong.
Example
Version 1 of a file has id and price (in dollars).
| Change | Effect on an existing consumer |
|---|---|
v2 adds an optional currency field | Safe: it can ignore the new field |
v3 renames price to amount | Breaks: the field it reads has gone |
v4 keeps price but now stores cents (1999 instead of 19.99) | Breaks silently: the type is unchanged and every figure is 100 times too large |
The third is why meaning, units and time semantics belong in a Data Contract, not only in the schema.
Common misunderstandings
- “If the types still match, the change is safe.” A changed unit or definition passes every type check.
- “Version numbers solve it.” A version number only helps if there are rules for what may change between versions and a way for readers to find out.
- “Just add a column.” Adding is usually safe, but a new required field breaks every writer that does not know it.
References
- Martin Kleppmann. Designing Data-Intensive Applications, chapter 4: Encoding and Evolution . O'Reilly Media, 2017. ISBN 978-1449373320.
- Apache Avro Specification, Schema Resolution . Apache Software Foundation.