Calibrate predicted probabilities on held-out data

A probability of 0.20 should mean something stronger than “this score ranks above another score.” Across comparable predictions near 0.20, roughly one fifth should be positive if the model is calibrated for that population. Post-hoc calibration learns a mapping from model scores to observed frequencies, so it needs data separate from base-model fitting.

Assign every split one role

Our synthetic workflow uses five authored periods. A 160-tree random forest fits on 3,000 training rows. A sigmoid calibrator then fits on a disjoint 1,000-row calibration period while the forest is frozen. The 1,500-row validation period is available for later policy choices, and the 1,500-row test period remains final. A fifth period represents a policy change.

The CalibratedClassifierCV documentation supports calibrating an already fitted model through FrozenEstimator and states that the user must keep model-fitting and calibration data disjoint.

On the calibration rows, the sigmoid moves mean probability from 6.39% to 5.40%, matching the 5.40% prevalence almost exactly. Yet Brier score worsens from 0.04777 to 0.04872 and log loss from 0.18373 to 0.19012. Matching the average rate does not guarantee better individual probabilities.

The later evidence is also unfavourable. On validation, sigmoid Brier score is 0.06407 versus 0.06253 uncalibrated. On test, it is 0.05407 versus 0.05401; log loss is 0.21116 versus 0.20854. We retain this negative result because a calibration method must earn selection rather than receive credit from its name.

from imbalance_cases import calibration_case

r = calibration_case()
assert r['fit_roles']['base'] == 'train'
assert r['fit_roles']['sigmoid'] == 'calibration'
assert r['fit_roles']['final'] == 'test'
test = r['metrics']['test']
assert test['sigmoid']['brier'] > test['uncalibrated']['brier']
assert test['sigmoid']['log_loss'] > test['uncalibrated']['log_loss']
print(r)

Run it in the imbalanced-model lab. The code rebuilds the forest and sigmoid mapping from fixed periods. It does not choose a method after inspecting test labels.

Treat calibration as model selection

Compare the uncalibrated score with predeclared sigmoid, isotonic or other supported mappings on validation data. Isotonic is flexible and can overfit small calibration sets. Sigmoid is smoother but can miss complex distortions. Cross-fitted calibration can use data more efficiently, provided all preprocessing and model selection remain inside the folds.

Use proper scores such as log loss and Brier score alongside reliability tables. Check relevant slices and time periods, because aggregate calibration can hide offsetting errors. A method chosen on validation should receive one final test assessment with uncertainty.

Calibration also expires. Changed prevalence, underwriting, review policy or data collection can break the mapping even when rank order remains useful. Monitor prediction means, observed rates after label maturity and reliability by stable bins.

Exercise: add an isotonic candidate using only the calibration period, choose between uncalibrated, sigmoid and isotonic on validation Brier score, and evaluate the single choice on test. Record why selecting again after seeing test performance would contaminate the final estimate.

NeuraPath's Data Science course emphasizes honest negative results and split roles. Calibration is a fitted model layer, so it deserves the same validation discipline as the classifier beneath it.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Class weights versus resampling in a controlled comparison.
• Continue with Reliability diagrams: read a calibration curve correctly.