Monitor calibration after a business-policy change

A business policy can change who receives an action, which outcomes are observed and the relationship between scores and labels. A calibration map fitted before the change may no longer represent the post-policy population even if the model code is identical.

Compare periods and policy slices

Our synthetic generator creates a known teaching mechanism. In the post-policy period, risk mix shifts upward and 493 of 1,000 high-risk rows receive an authored reduction in target logit. The fitted model does not use the policy flag.

Before the change, the 1,500-row test period has prevalence 6.33%, mean predicted probability 5.38%, Brier score 0.05407 and log loss 0.21116. After the change, prevalence is 6.00%, mean prediction rises to 7.61%, Brier score to 0.05723 and log loss to 0.21581.

Within the post-policy period, untreated rows have prevalence 3.55% and mean prediction 3.72%. Treated rows have prevalence 8.52% and mean prediction 11.62%. Their Brier score, 0.08098, is also much higher than 0.03413 for untreated rows.

from imbalance_cases import policy_case

r = policy_case()
assert r['periods']['test']['policy_applied_rate'] == 0
assert r['periods']['post_policy']['policy_applied_rate'] == .493
assert r['periods']['post_policy']['mean_probability'] > r['periods']['post_policy']['prevalence']
assert r['post_policy_slices']['1']['brier'] > r['post_policy_slices']['0']['brier']
print(r)

Reproduce the comparison in the imbalanced-model lab. Because the generator declares the mechanism, we know why this synthetic shift occurred. Real monitoring usually observes a change without identifying its cause.

The scikit-learn calibration guide explains reliability analysis. Post-change monitoring should preserve counts, label maturity and the exact model-policy version beside every metric.

Monitor in a causal system without claiming causality

An action targeted by model score creates selection. Outcomes for treated and untreated rows differ in baseline risk as well as possible treatment effect, so comparing their raw event rates does not estimate policy impact. Randomized rollout, a valid quasi-experimental design or stronger causal assumptions are needed for that question.

For model monitoring, compare score distributions immediately and outcome metrics after labels mature. Track Brier score, log loss and reliability bins by policy eligibility and action received. Distinguish population drift, calibration drift, target-definition changes and selective labels.

Predeclare alert thresholds and response: investigate data, pause automation, recalibrate on a valid post-change sample or retrain. Do not automatically recalibrate on selectively observed outcomes. Retain an audit sample and log counterfactual eligibility where policy permits.

Exercise: create three dashboards: score-only early warning, mature calibration, and policy-effect evaluation. List the fields, latency and causal claims allowed in each. Show why combining them into one chart would invite an unsupported conclusion.

NeuraPath's Data Science course treats deployed models as parts of changing decision systems. Calibration monitoring is strongest when it can separate what changed from what the data can actually prove.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Document the trade-off between automation and manual review.
• Return to the cluster foundation in Class imbalance: distinguish prevalence from model difficulty.