Data validation for model inputs with realistic edge cases

Input validation should test failures that can occur in the real pipeline: duplicate identities, non-finite values, impossible counts, unfamiliar categories and extreme but potentially legitimate values.

Give every rejection a reason

Our six-row teaching batch contains one valid row and five edge cases. The validator accepts only ok. It rejects negative count, NaN amount, an amount above the illustrative review limit, unknown channel and a repeated row ID.

The extreme amount receives amount_above_review_limit, not “invalid.” A high value may be genuine; the contract routes it for review because the model was not validated there.

from feature_cases import validation_case

r = validation_case()
assert r['accepted'] == 1 and r['rejected'] == 5
cases = {row['case']: row for row in r['cases']}
assert cases['ok']['errors'] == []
assert cases['nan_amount']['errors'] == ['nonfinite_amount']
assert cases['duplicate_id']['errors'] == ['duplicate_id']
print(r)

Run all cases in the feature-engineering lab. These are reference behaviours for one illustrative policy.

Separate invalid, unsupported and suspicious

Invalid values violate syntax or domain logic, such as a negative count. Unsupported values may be legitimate but outside model evidence, such as a new category. Suspicious values deserve review but should not be deleted automatically. Each class needs a distinct response.

Validate row grain and batch invariants as well as fields. Duplicate IDs, join multiplication, unexpected row loss and shifted category frequency can pass type checks. Reconcile input and output counts.

Run the same contract in offline training and online serving. Save rejection counts and reason codes without exposing sensitive payloads. Alert on rate changes by source, because a surge often signals an upstream release.

Test boundary values, Unicode, timezone changes, empty batches, enormous batches and serialization differences. Property-based tests can expand coverage, while curated business cases preserve known failures.

Exercise: add null-category, negative-zero, duplicate-with-conflicting-values and maximum-boundary cases. Define reject, review and fallback actions, then prove batch row reconciliation under each path.

NeuraPath's Data Science course connects validation rules to operational responses. A useful contract explains why a row failed and what happens next.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design a feature dictionary with owners and units.
• Continue with Handle outliers before deciding to cap or remove them.