Measure false-positive workload before deploying a classifier

A false-positive rate is a fraction of actual negatives. Operations receive a count of alerts. Before deployment, connect the fraction to expected volume, batch timing, duplicate handling and analyst capacity.

Convert the frozen test result into scenarios

Our validation-selected threshold is 0.25148. On 1,500 synthetic test transactions it produces 55 false positives, a false-positive share of total rows equal to 3.67%. Because this article projects queue volume from total incoming rows, it uses 55/1500. The conventional false-positive rate conditional on actual negatives would instead use 55/1405; the denominator must be named.

Holding the observed per-row false-positive share fixed gives these arithmetic scenarios:

These are linear projections, not forecasts. They assume the score distribution, prevalence, case mix, threshold and label definition remain like the test period.

from imbalance_cases import workload_case

r = workload_case()
assert r['observed_test_false_positives'] == 55
assert r['test_rows'] == 1500
assert abs(r['false_positive_rate'] - 55/1500) < 1e-12
assert r['projected_false_positives'][-1]['expected_at_fixed_rate'] == 50000*55/1500
print(r)

Run the calculation in the imbalanced-model lab. The model outcome is synthetic, and review effectiveness is not part of this calculation.

Build the complete queue equation

False positives are only part of selected workload. Analysts see true positives too, plus duplicates, reopened cases and escalations. Start from total selected cases at the threshold, then map transactions to the actual unit of work. Five alerts for one account may become one case or five separate investigations.

Capacity needs a time denominator: per hour, shift or day. Add arrival peaks, analyst schedules, service-time distribution, backlog and service-level objective. Average daily capacity can still fail during bursts. A queue simulation may be more useful than a single multiplication.

Attach uncertainty to the test rate and create scenarios for population shift. If prevalence rises, the false-positive share may fall while total selected workload rises. If a feature pipeline changes, both score distribution and error rates can move. Monitor incoming volume and score histograms before mature labels arrive, then update error estimates when outcomes become observable.

Do not optimize the threshold on test workload after using test labels for final evaluation. If the queue is unacceptable, revise the development policy and collect a new untouched assessment period.

Exercise: add true-positive reviews, a 15% duplicate-account collapse rate and review times drawn from a documented distribution. Simulate hourly backlog under normal and peak arrival volumes, then state which inputs are measured and which are assumptions.

NeuraPath's Data Science course turns classification metrics into operational quantities. The useful workload estimate names its denominator, time window and assumptions.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Prior probability shift: when yesterday's probabilities mislead.
• Continue with Threshold selection with unequal error costs.