Prior probability shift: when yesterday's probabilities mislead

Prior probability shift means the class prevalence changes while the feature distribution within each class stays the same. A model can retain similar ranking under that restricted shift while its old probability scale becomes wrong for the new base rate.

Create a shift that matches the assumption

Our source test period has 6.33% positives. We form a 1,000-row teaching sample by drawing 150 positive and 850 negative rows with replacement from that period. The resulting prevalence is exactly 15%, while the sampled feature-score relationship within each class comes from the same source rows.

Raw model probabilities average 7.47%, well below the constructed 15% rate. Brier score is 0.11384 and log loss 0.39329.

Let source prevalence be pi_old and target prevalence pi_new. Convert each old probability to odds, multiply by the ratio of new prior odds to old prior odds, and convert back:

new_odds = old_odds * [pi_new/(1-pi_new)] / [pi_old/(1-pi_old)].

After correction, mean probability is 14.97%, Brier score 0.10697 and log loss 0.35380. The result works here because the fixture was constructed to satisfy pure prior shift.

from imbalance_cases import prior_shift_case

r = prior_shift_case()
assert abs(r['source_prevalence'] - 95/1500) < 1e-12
assert r['shifted_prevalence'] == .15
assert abs(r['prior_corrected']['mean_probability'] - .15) < .001
assert r['prior_corrected']['log_loss'] < r['raw']['log_loss']
print(r)

Reproduce the resampling and odds calculation in the imbalanced-model lab. This controlled case is not evidence that real drift is prior shift.

Test the assumption before correcting

In practice, a prevalence change can accompany new fraud tactics, customer mix, instrumentation, policy or label definitions. Then class-conditional feature distributions may change and the simple correction can remain wrong. Labels may also arrive late, making the apparent current prevalence biased.

Estimate the new prior from mature, representative outcomes or a validated shift-estimation method. Compare feature and score distributions within class once labels mature. Check performance by time and segment. If the mapping from features to outcomes changed, retraining, new features or a redesigned target may be necessary.

Correction changes probabilities and therefore every probability threshold. Revalidate queue size, precision, recall and cost after applying it. Preserve the old model, estimated prior, correction formula and effective date in the model record so decisions can be reconstructed.

Exercise: repeat the correction after changing the positive-class score distribution as well as prevalence. Show a case where the corrected mean matches 15% but reliability bins remain wrong. Explain why aggregate mean agreement is insufficient.

NeuraPath's Data Science course connects drift labels to testable assumptions. Prior correction is powerful when its narrow condition holds and hazardous when a broader mechanism shift is hidden behind the same prevalence change.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate rare-event models with confidence intervals.
• Continue with Measure false-positive workload before deploying a classifier.