Brier score versus log loss for probability forecasts

Brier score and log loss both evaluate probability forecasts against binary outcomes. They are proper scoring rules: in expectation, honest probabilities minimize the score. Their penalty shapes differ, so a result should name the score rather than saying a model has “good probability accuracy.”

Compute both scores on three exact forecasts

Use two outcomes, [1, 0]. Three probability vectors make the difference visible:

Brier score is the mean squared difference between probability and outcome. Log loss uses the negative logarithm of probability assigned to the observed class. As a forecast approaches zero for an event that occurs, log loss grows sharply. That makes confident mistakes especially influential.

The scikit-learn metrics reference documents brier_score_loss and log_loss. Both require probabilities on the intended population; neither tells you which threshold an operation should use.

from imbalance_cases import score_case

r = score_case()
cases = r['two_row_cases']
assert abs(cases['moderate']['brier'] - .16) < 1e-12
assert cases['one_confident_error']['log_loss'] > 2.3
assert cases['perfectish']['brier'] < .001
print(r)

Run this in the imbalanced-model lab. The two-row construction explains the formula; it is too small to compare real models.

Interpret the rare-event reference honestly

On the 1,500-row synthetic test period, the uncalibrated forest has Brier score 0.05401 and log loss 0.20854. The sigmoid-calibrated version has 0.05407 and 0.21116. Both scores are slightly worse after calibration in this period.

These values depend strongly on prevalence and case mix. A constant forecast near the base rate can achieve a low Brier score in a rare-event dataset while providing little ranking value. Always compare with a simple probability baseline, report prevalence and add discrimination metrics when ranking matters.

Log loss requires finite probabilities; robust implementations clip exact zeros and ones internally or by declared policy. Silent clipping changes the maximum penalty, so record it when comparing systems. Sample weights also change the estimand and must match the deployment population.

Use score differences with uncertainty rather than judging isolated decimals. Inspect which rows dominate log-loss changes and whether they expose data errors, unfamiliar regions or true overconfidence. Do not remove difficult cases merely because they hurt the score.

Exercise: create a 1,000-row set with 1% positives. Compare a constant 1% forecast, a perfect ranking with poorly scaled probabilities and a weaker ranking with calibrated probabilities. Report Brier, log loss, average precision and ROC AUC, then explain why no single number answers every use case.

NeuraPath's Data Science course connects scoring rules to the decision a probability will support. Exact formulas, baselines and population context make the comparison meaningful.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Reliability diagrams: read a calibration curve correctly.
• Continue with Choose a fraud-review threshold from analyst capacity.