Design a training dataset contract

A schema lists columns and types. A training contract also defines row grain, uniqueness, timestamps, units, allowed values, freshness and ownership. These semantics determine whether a feature means the same thing across training and serving.

Execute five core checks

The data-engineering lab defines one row per event_id, UTC event-time convention, integer cents, three allowed event types and a 24-hour freshness objective.

from data_engineering_cases import contract_case

result = contract_case()
assert result["rows"] == 500
assert result["passed"] is True
assert all(result["checks"].values())
print(result["contract"]["grain"], result["checks"])

The fixture contains required columns, unique keys, nonnegative amounts, allowed event types and ingestion timestamps no earlier than event time.

Add model semantics

State target definition, prediction timestamp, observation window and exclusion rules. Distinguish event time from ingestion and update time. Define how corrections and deletions propagate. For snapshots, specify one row per entity and as-of date.

Units belong in the contract. Changing cents to rupees without a schema change can scale a feature by 100 while passing type checks. Categorical definitions can drift just as silently.

Assign ownership and compatibility

Name the source and consumer owners, change notice, supported versions and incident response. Classify additive, breaking and semantic changes. Run contract checks at ingestion and again before training because intermediate transformations can introduce errors.

The Data Science course connects contracts to feature lineage and deployment validation.

Exercise

Extend the contract with target availability, privacy class, retention, late-event policy and owner. Inject five violations and verify that publication stops with useful errors.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in ETL versus ELT through a model-training pipeline.
• Continue with Batch versus streaming data for an actual prediction need.

Reference: TensorFlow Data Validation schema concepts.