Isolation forests with an analyst-review budget

An anomaly model produces a continuous ranking. Operations need a cutoff, often defined by how many cases analysts can review. A contamination parameter or zero score threshold should not silently become the business policy.

Compare three queue sizes

We fit a 200-tree Isolation Forest without labels on 1,020 synthetic rows and rank descending anomaly score. The fixture contains 20 generator-known shifted points for after-fit evaluation.

Doubling the budget from 20 to 40 finds the last authored anomaly but adds 19 more normal rows. The preferred queue depends on review cost and the value of the missed case, neither of which the anomaly algorithm knows.

from unsupervised_cases import anomaly_case

r = anomaly_case()
by_budget = {row['budget']: row for row in r['budgets']}
assert by_budget[10]['anomaly_recall'] == .5
assert by_budget[20]['review_precision'] == .95
assert by_budget[40]['review_precision'] == .5
print(r)

Run the ranking in the unsupervised lab. The scikit-learn outlier benchmark likewise evaluates unsupervised scores with labels after fitting.

Choose rank, threshold or hybrid deliberately

A top-k policy fixes workload but lets the score cutoff change each batch. A fixed score threshold holds the anomaly standard but allows volume to fluctuate. A hybrid can cap the queue while enforcing a minimum score. Specify batch size, timing, tie handling and duplicate consolidation.

Select the policy on a labelled validation period or through a reviewed pilot. Evaluate once on a later period. If labels exist only for reviewed cases, precision is observable but recall is not; create random audit samples or other outcome coverage.

Monitor feature distributions, score quantiles, selected counts, review dispositions and time to resolution. A sudden rise in alerts may reflect data pipeline changes. Preserve the threshold and score version needed to reconstruct each queue.

Exercise: assign review and miss costs to the three budgets, then test a fixed-score policy on a shifted period. Report both per-row and per-account queues when several rows belong to one entity.

NeuraPath's Data Science course connects anomaly ranking to capacity and evidence. The review budget is part of the model policy, not an afterthought.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Anomaly detection: validate alerts before calling them fraud.
• Continue with Detect unstable customer segments after retraining.