Reliability diagrams: read a calibration curve correctly

A reliability diagram groups probability predictions and compares each group's mean forecast with its observed positive rate. A point on the diagonal is locally consistent with calibration. The visual can still mislead when bins have different support, intervals are wide or labels are immature.

Carry counts beside every point

Our test example uses sigmoid-calibrated probabilities for 1,500 synthetic rows. Equal-count binning creates five groups of 300 rows:

The sample-weighted absolute difference between bin means and observed rates is 0.01878. That summary depends on these exact bins. It is descriptive, not a calibration hypothesis test.

The scikit-learn calibration guide defines calibration curves and notes that the accompanying probability histogram helps reveal where predictions concentrate. The calibration_curve function supports uniform-width and quantile binning.

from imbalance_cases import reliability_case

r = reliability_case()
assert len(r['bins']) == 5
assert sum(row['n'] for row in r['bins']) == 1500
assert sum(row['positives'] for row in r['bins']) == 95
assert abs(r['weighted_absolute_gap'] - .018782888405165635) < 1e-12
print(r)

Run the calculation in the imbalanced-model lab. It forms each interval from the test probabilities and recomputes event rates; no chart coordinates are copied by hand.

Read the diagram as an estimate with design choices

Equal-width bins make probability ranges comparable but can leave high-score bins nearly empty. Equal-count bins stabilize row counts but create intervals with very different widths, as the broad top interval shows. Changing the number or type of bins can change the apparent curve.

Include row and positive counts, confidence or uncertainty intervals where appropriate, and a probability histogram. Avoid declaring a four-event bin “well calibrated” or “badly calibrated” from its point estimate alone. Check whether repeated entities require clustered uncertainty.

Aggregate points can hide segment errors. One group may be underpredicted and another overpredicted inside the same bin. Slice only on predeclared, meaningful dimensions and retain sample-size context. When labels arrive late, build bins from an as-of cohort with mature outcomes; treating unresolved cases as negatives creates artificial overprediction.

A reliability plot evaluates forecast behaviour under the observed population. It does not measure ranking quality, recommend a decision threshold or prove that changing a feature will change risk.

Exercise: recompute this table with five equal-width bins. Report empty bins, interval widths and positive counts. Then bootstrap accounts within each fixed bin and explain what uncertainty source that interval still omits.

NeuraPath's Data Science course teaches calibration plots with the table beneath them. The counts and binning rule are part of the result, not decoration.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Calibrate predicted probabilities on held-out data.
• Continue with Brier score versus log loss for probability forecasts.