Data science interview: diagnose leakage in a suspicious score

A near-perfect score is a reason to investigate before celebrating. In many business tasks, it can reveal a post-outcome field, duplicates across splits, target-derived preprocessing or a validation set that influenced model selection.

An interview answer should trace time and information rather than merely say “check for leakage.”

Reconstruct the prediction moment

The career evidence lab gives an invented account-closure example.

from career_cases import leakage_diagnosis_case

result = leakage_diagnosis_case()
assert result["reported_auc"] == 0.99
assert result["suspect_features"] == ["account_closed_reason"]
assert result["retest_auc_without_future_field"] == 0.71
assert result["suspicious_score_rejected"] is True

account_closed_reason exists only after the outcome, so it cannot support an earlier closure-risk decision. Removing it lowers the authored AUC from 0.99 to 0.71. These numbers demonstrate the diagnostic sequence on a fixture; they are not operating model performance.

Audit four leakage routes

1. 1Feature time: Was every value available before the prediction cutoff? Recreate features from timestamped source events.
2. 2Entity overlap: Do the same people, devices or near-duplicate records cross splits? Hash and group identifiers before splitting.
3. 3Preprocessing: Were imputation, scaling, vocabulary, feature selection or target encoding fit only on training data and folds?
4. 4Selection: Did test labels influence model, threshold or feature choices?

Also compare the score with a naive baseline and a stripped-down feature set. Train one-feature models to find suspicious proxies. Inspect missingness: a field populated only after manual investigation can encode the decision even if its name appears harmless.

Preserve the failed result

Do not delete the 0.99 experiment. Mark it invalid, record the leaking feature and add a regression test that enforces feature-time eligibility. The contrast with the corrected evaluation is strong portfolio evidence because it shows that you rejected an attractive result.

Explain the business consequence. Leakage makes offline evaluation optimistic, which can waste review capacity, trigger harmful actions or fail after deployment when future fields are absent.

The Data Science course ties leakage diagnosis to point-in-time joins, pipeline boundaries and model cards.

Exercise

Create a table with feature name, source timestamp, availability timestamp, prediction cutoff and reason for inclusion. Add one post-outcome feature and one duplicated entity, confirm the score changes, then make both failures executable tests.

Continue learning

This article is part of the Data science careers and portfolio decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Explain model validation in a data science interview.
• Continue with Data science interview: choose a metric from business costs.

Reference: scikit-learn leakage guidance.