Confusion matrices at more than one threshold

A probability model and its classification threshold make separate decisions. The model produces a score; the threshold determines which cases receive a positive label or enter an action queue. Compare thresholds on development evidence and keep the final test assessment separate.

Our synthetic inactivity model has eighty February validation snapshots, including 47 positives. The original experiment prespecified threshold 0.5. This article examines 0.3, 0.5 and 0.7 on those same validation predictions to explain the tradeoff; it does not replace the frozen test result.

Read the counts before the percentages

Lowering the threshold from 0.5 to 0.3 finds 21 additional positives and selects nineteen additional negatives. The queue grows from 24 to 64 cases. Raising it to 0.7 selects nobody in this validation set.

These results come from the fixed original snapshot simulation. They are not estimates of the effect of contacting an account, and inactivity is not the same as cancellation.

Reproduce the threshold sweep

import numpy as np
from metric_cases import validation_thresholds

rows = validation_thresholds()
expected = [[9,24,7,40],[28,5,28,19],[33,0,47,0]]
for row,counts in zip(rows,expected):
    assert [row[k] for k in ['tn','fp','fn','tp']]==counts
    assert sum(counts)==80
assert np.isclose(rows[0]['recall'],40/47)
assert np.isclose(rows[1]['precision'],19/24)
assert rows[2]['precision'] is None
assert len({row['log_loss'] for row in rows})==1
print(rows)

Run from the evaluation lab. The probability-based log loss stays unchanged because the underlying probabilities stay unchanged. Confusion counts change because the classification rule changes.

The rule uses probability >= threshold, so a score exactly equal to the threshold is positive. Record this tie convention, especially when scores are rounded or a capacity rule selects cases near a boundary.

Choose a threshold from a declared decision

At 0.3, precision is 62.5% and recall about 85.11%. At 0.5, precision is about 79.17% and recall 40.43%. A stakeholder who can review only thirty cases faces a different problem from one who must identify most positives regardless of queue size.

Supply false-positive and false-negative consequences, review capacity, and any mandatory constraints. Do not call the threshold with the highest accuracy “optimal” without specifying the objective and allowed workload.

Scikit-learn's precision–recall curve reference describes how classification behavior changes across score thresholds. The three-point table here preserves the original project's exact validation counts.

Avoid two common interpretation errors

First, precision does not have to improve monotonically at every higher threshold on a finite dataset. Higher thresholds cannot increase the number of selected positives or negatives for fixed scores, but the selected mixture can change irregularly.

Second, a threshold chosen from validation data has been tuned. If many thresholds or cost assumptions are explored, record that search. Do not subsequently advertise performance on the same validation data as an untouched final assessment.

The original March test at threshold 0.5 still has recall40%. This article does not claim that a newly chosen threshold has been independently validated for deployment.

Exercise: assume a validation review capacity of thirty cases. Inspect distinct score thresholds, report feasible choices and their missed-positive counts, and freeze a selection rule before obtaining a new assessment set. State whether capacity is per batch, day or account.

NeuraPath's Data Science course connects classification outputs with practical operating decisions. A good threshold report shows the queue and errors that the decision actually creates.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Classification metrics when accuracy hides failure.
• Continue with Precision-recall versus ROC curves for rare outcomes.