Schema evolution without silently changing feature meaning

A pipeline can keep running after a schema change and still become wrong. The dangerous case is a field that retains its name and type while its business meaning changes. If amount_cents starts carrying rupees, every record remains a valid integer while a spend feature changes by a factor of 100.

Schema evolution therefore needs two reviews: can the reader parse the new data, and does the data still mean what the model contract says?

Classify the change before accepting it

The local data-engineering lab records three representative changes.

from data_engineering_cases import schema_evolution_case

result = schema_evolution_case()
assert result["breaking_changes"] == 2
assert result["silent_unit_change_allowed"] is False
print(result["changes"])

Adding a nullable source_channel can be backward compatible at the storage layer. It still needs an owner, a null policy and a decision about whether historical rows require a backfill. Renaming amount_cents to amount is structurally breaking because consumers referencing the old field fail. Changing the unit under the existing name is semantically breaking because consumers may not fail at all.

Treat compatibility as a matrix rather than one boolean:

Put semantics in the contract

A useful field contract includes type, nullability, unit, timezone, allowed values, event-time meaning, owner and effective date. For a derived feature, also store the source fields, filters, lookback window and cutoff convention. spend_30d is incomplete documentation unless it says whether refunds count and whether the interval includes the prediction timestamp.

When a producer proposes a change, identify downstream datasets and model versions through lineage. Run old and new transformations over an overlap window. Compare row counts, null rates, units, feature distributions and a small set of hand-calculated records. Publish a new version only after the model owner accepts the semantic diff.

Make incompatible changes observable

Do not overwrite a versioned dataset and hope monitoring catches the result. Write the new output separately, attach its contract and validation report, then move an approved pointer. Keep the previous version available for rollback according to retention policy.

The Data Science course connects this data contract to point-in-time features, model evaluation and deployment checks. The assessment evidence should show both a schema test and a meaning test.

Exercise

Take a feature table with amount_cents, plan and event_time. Propose one additive, one structural and one semantic change. For each, name affected consumers, create a compatibility test, define a migration window and state which model artifacts must be reevaluated.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data quality checks at ingestion and before training.
• Continue with Create a backfill plan with bounded impact.

References: Apache Parquet logical types and JSON Schema object validation.