Class imbalance: distinguish prevalence from model difficulty

Class imbalance describes how often each label occurs. It does not, by itself, tell you whether the classes are easy to separate. A rare target can have a clear signal; a balanced target can be nearly unpredictable. Mixing prevalence, ranking quality and threshold decisions leads to misleading model claims.

Read four different questions separately

Our synthetic test period contains 1,500 transactions and 95 positives, a prevalence of 6.33%. A classifier trained on an earlier 3,000-row period produces ROC AUC 0.7799 and average precision 0.2411 on this test set.

At the default probability threshold of 0.5, however, it predicts only seven positives: four true positives and three false positives. Precision is 57.14%, while recall is just 4.21%. The ranking has useful signal, but the default operating point retrieves very few actual positives.

An always-negative rule achieves 93.67% accuracy because 93.67% of rows are negative. That high number says nothing about finding the rare class. It is a prevalence result disguised as model performance.

The scikit-learn precision-recall example recommends precision-recall analysis for heavily imbalanced problems and defines average precision from operating points. ROC AUC, average precision and a thresholded confusion matrix answer different questions.

from imbalance_cases import prevalence_case

r = prevalence_case()['splits']['test']
assert r['n'] == 1500 and r['positives'] == 95
assert abs(r['prevalence'] - 95/1500) < 1e-12
assert abs(r['all_negative_accuracy'] - 1405/1500) < 1e-12
assert r['roc_auc'] > .77
assert r['recall'] < .05
print(r)

Run the example in the imbalanced-model lab. The fixture has repeated synthetic accounts and authored chronological periods. It is not a real fraud benchmark.

Diagnose the source of difficulty

Start with positive support and prevalence in each split. Then inspect score distributions, precision-recall operating points and uncertainty. A low average precision may reflect overlapping features, label error, population shift or few positives. Resampling cannot invent missing information.

Threshold choice belongs to the action. A review queue with 75 slots requires a different operating point from an automatic block with costly false positives. Report the confusion matrix and absolute workload at the selected threshold. A false-positive rate that looks small can still create thousands of cases at production volume.

Prevalence also changes the interpretation of probability forecasts. If the base rate moves while class-conditional feature distributions stay fixed, an old probability scale can become wrong even when ranking remains stable. If the feature-to-label mechanism changes too, a simple prior correction is insufficient.

Do not “solve imbalance” as a generic preprocessing step. State the real failure: missed positives, poor probabilities, excessive review load, unstable subgroup results or weak labels. That statement determines the metric, split and intervention worth testing.

Exercise: keep a fixed ranking of 1,000 scores and create evaluation sets with 2%, 10% and 30% positive prevalence by stratified resampling. Compare ROC AUC, average precision, precision at a fixed threshold and the always-negative accuracy. Explain which quantities change and why.

NeuraPath's Data Science course connects imbalanced learning to decisions, calibration and reproducible evaluation. The first step is naming whether the problem is rarity, separability or the operating policy.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with SMOTE inside cross-validation without leakage.
• Then apply it in Class weights versus resampling in a controlled comparison.