Evaluate a classifier when labels arrive late

Many outcomes are not known at prediction time. Fraud confirmation, churn, default and returns can mature days or months later. Treating every unresolved row as negative makes a recent model look artificially precise or poorly calibrated, depending on the delay pattern.

Join outcomes as they were known at a cutoff

Our synthetic exercise examines the latest 90 test decisions. The evaluation cutoff is three days after the final decision. Every row stores decision day and label-availability day.

At the cutoff, 84 labels are mature and six are unresolved. The mature subset contains seven positives. Because the generator knows the future, we can see that the six unresolved rows happen to be negative; a real as-of pipeline would not know this and must not use it.

On all 90 labels viewed retrospectively, average precision is 0.4281. On the 84 labels available at the cutoff, it is 0.4325. The difference is small in this seed, but the contract matters regardless of how much the metric changes.

from imbalance_cases import delay_case

r = delay_case()
assert r['cohort_rows'] == 90
assert r['mature_rows'] == 84 and r['immature_rows'] == 6
assert r['mature_rows'] + r['immature_rows'] == r['cohort_rows']
assert r['as_of_cutoff']['n'] == 84
print(r)

Reproduce the cutoff join in the imbalanced-model lab. Future targets are used only to audit the synthetic construction, not to form the as-of metric.

Separate maturity bias from model error

If positives take longer to confirm than negatives, the mature subset is not representative of the full cohort. Simply dropping unresolved rows can still bias an early estimate. Report maturity rate by score band, predicted class, segment and eventual outcome once history becomes available.

Use fixed cohorts and evaluation lags. For example, publish January predictions only after a declared 60-day outcome window, while showing an explicitly provisional dashboard sooner. Never compare a fully mature old model with an immature new model as though the labels were equivalent.

Store prediction timestamp, model version, feature-as-of time, outcome event time, label-availability time and revision history. A label can be corrected after first arrival; decide whether dashboards reconstruct historical knowledge or use latest truth.

Survival or censoring methods may help when time-to-event is the target, but they require their own assumptions. They do not justify silently calling censored observations negative.

Exercise: change the synthetic delay so positives mature much later. Plot apparent prevalence and average precision at lags of 3, 7, 14 and 30 days. Compare each provisional estimate with the mature cohort and explain which dashboard labels would prevent misreading.

NeuraPath's Data Science course treats label availability as part of the data model. An evaluation is reproducible only when it can answer “known as of when?”

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a synthetic fraud benchmark with transparent assumptions.
• Continue with Document the trade-off between automation and manual review.