Choose a fraud-review threshold from analyst capacity

A review team acts on cases, not abstract rates. If a batch contains 1,500 transactions and analysts can inspect at most 75, the threshold must respect that queue before model metrics become operationally relevant.

Optimize the validation decision under a hard limit

Our synthetic model produces probabilities for 1,500 validation rows. We consider every distinct score as a candidate threshold and retain candidates selecting between one and 75 rows. Among those, we maximize the number of true positives; ties prefer the higher threshold.

The chosen threshold is 0.25148. It selects 62 validation cases: 25 true positives and 37 false positives. Validation precision is 40.32% and recall is 22.12%. The algorithm does not fill all 75 slots because lowering to the next score would not improve the lexicographic rule within the candidate set.

We then freeze the threshold and apply it once to 1,500 test rows. It selects 78 cases: 23 true positives and 55 false positives. Test precision is 29.49% and recall is 24.21%.

Why did the test queue exceed 75? Capacity was defined as 75 per 1,500 validation batch, but a fixed probability threshold does not guarantee the same count in a new batch. The score distribution changed.

The scikit-learn cost-sensitive threshold example separates probability estimation from the decision cutoff. Our example adds a hard queue constraint.

from imbalance_cases import threshold_case

r = threshold_case()
assert r['capacity_per_batch'] == 75
assert r['chosen_validation']['selected'] == 62
assert r['chosen_validation']['selected'] <= r['capacity_per_batch']
assert r['test_at_fixed_threshold']['tp'] == 23
assert r['test_at_fixed_threshold']['fp'] == 55
print(r)

Reproduce it in the imbalanced-model lab. The target and review process are synthetic; a positive label does not prove that review prevents loss.

Decide whether capacity applies to rank or probability

A top-75 policy guarantees batch workload but gives the last selected case a changing probability cutoff. A fixed threshold preserves an evidential standard but allows queue size to vary. A hybrid can require both a minimum score and a maximum rank, with an explicit overflow rule.

Specify the batch clock, analyst availability, carryover, duplicates and service-level deadline. Repeated accounts may generate several alerts; an account-level queue can differ from a transaction-level queue. Tie-breaking must be deterministic and auditable.

Threshold selection should use validation data or nested resampling. Test data estimate the frozen policy once. After deployment, monitor score distribution, selected counts, mature precision and review outcomes. If capacity changes, revalidate the policy rather than silently moving the threshold.

Exercise: implement fixed-threshold, top-75 and hybrid policies on validation and test. Compare workload, overlap and retrieved positives. Define what happens when 20 cases tie at the final rank and when an account produces five transactions in one batch.

NeuraPath's Data Science course treats capacity as part of model design. A usable threshold states what is fixed, what may vary and how the queue behaves when the next period differs.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Brier score versus log loss for probability forecasts.
• Continue with Optimize recall subject to a minimum precision constraint.