Detect label noise in an imbalanced dataset

Rare-event labels often come from investigations, chargebacks, delayed outcomes or rules that change over time. A wrong label can strongly affect a small positive class. Model disagreement can prioritize records for review, but it does not prove that a label is wrong.

Use generator truth to test the audit code

Our synthetic fixture stores two targets. target is the clean generated outcome. noisy_target flips positives with probability 12% and negatives with probability 0.8%. A generator-only flag records which rows were actually changed. None of these noise fields enters the fitted model.

In the 3,000-row training period, 47 labels are flipped. Of 205 clean positives, 12.20% flip; of 2,795 negatives, 0.79% flip. The observed noisy positive count is 202, only three below the clean count, because positive-to-negative and negative-to-positive changes partly cancel.

Across all periods, positive flip rates vary because support is finite. The test period contains 26 known flips; the clean and noisy positive totals are 95 and 99. Comparing only aggregate prevalence would miss most row-level corruption.

from imbalance_cases import label_noise_case

r = label_noise_case()
train = next(row for row in r['splits'] if row['split'] == 'train')
test = next(row for row in r['splits'] if row['split'] == 'test')
assert train['known_generator_flips'] == 47
assert train['clean_positives'] == 205 and train['observed_noisy_positives'] == 202
assert test['known_generator_flips'] == 26
print(r)

Run the audit in the imbalanced-model lab. The known flag exists only because the data are synthetic. A real audit lacks direct access to truth.

Build evidence independent of the model under review

Potential warning signals include duplicate records with conflicting labels, impossible event sequences, label timestamps before evidence exists, high reviewer disagreement and strong model-label disagreement across out-of-fold predictions. Each signal can also reflect a valid hard case or a data pipeline bug.

Review a blinded, stratified sample with documented adjudication. Oversample suspicious rows for diagnosis while retaining weights needed to estimate overall error rates. Record original labels, reviewer decisions, evidence, timestamps and reasons. Do not silently overwrite history.

Positive and negative label quality need separate estimates. A low negative flip rate can create many false positives when negatives vastly outnumber positives. Delayed labels are another category: unresolved is not negative. Define the maturity window before measuring noise.

Avoid training a model on all rows, using its in-sample confidence to “correct” labels and then reporting better performance on those corrected labels. That circular workflow rewards the model for rewriting disagreement. Use out-of-fold scores and independent review, then evaluate on untouched adjudicated data.

Exercise: rank the synthetic noisy rows by disagreement from an out-of-fold classifier without reading the flip flag. Measure precision among the top 25 reviews against generator truth, then explain why the same calculation would only be an estimate after human adjudication in real data.

NeuraPath's Data Science course treats label quality as a measured data process. Suspicion, review evidence and confirmed correction remain separate fields.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Tune multiclass thresholds without using the test set.
• Continue with Build a synthetic fraud benchmark with transparent assumptions.