Data quality checks at ingestion and before training

Ingestion checks protect the shared dataset; pretraining checks protect a particular model run. Both are needed because transformations and filters can create issues after raw data passes.

Inject three ingestion failures

The data-engineering lab duplicates an event ID, makes one amount negative and removes one event timestamp.

from data_engineering_cases import quality_case

result = quality_case()
assert result["ingestion_passed"] is False
assert result["pretraining_passed"] is True
print(result["ingestion_checks"])

Required columns and minimum row count still pass, but key uniqueness, timestamp completeness and nonnegative amount fail. A clean fixture passes the same pretraining checks.

Separate structural and statistical gates

At ingestion, check schema, key, type, range, referential integrity, event/ingestion ordering and freshness. Quarantine bad records or block publication according to a declared policy; do not silently drop them.

Before training, check population, target availability, feature cutoff, class prevalence, entity overlap, missingness and time window. Compare with a versioned reference but treat drift as evidence to investigate, not an automatic data deletion rule.

Make failures actionable

Each check needs severity, owner, expected/actual values and a response. Record denominators and rejected IDs under appropriate access. Prevent downstream training from reading an unapproved version.

The Data Science course places data-quality evidence before model metrics.

Exercise

Create ten checks split across ingestion and pretraining. Inject one fault at a time, verify messages and define block, quarantine or warning behavior with owners.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Airflow DAG design: separate orchestration from business logic.
• Continue with Schema evolution without silently changing feature meaning.

Reference: Great Expectations validation concepts.