Handle class imbalance in an image dataset

An image classifier trained on a majority class can minimize average loss by ignoring the minority. Class weighting changes how errors contribute to training, but it does not guarantee useful recall or a suitable decision threshold. Evaluate it against the action the minority prediction triggers.

A deliberately difficult result

The local lab trains on 80 vertical and 20 horizontal images, a 4:1 ratio, then evaluates on a later authored test set.

from deep_learning_cases import image_imbalance_case

result = image_imbalance_case()
assert result["training_counts"] == {"negative": 80, "positive": 20}
plain = result["models"]["unweighted"]
weighted = result["models"]["positive_weight"]
assert weighted["positive_recall"] > plain["positive_recall"]
print(plain["positive_recall"], weighted["positive_recall"])

Weighting moves recall above zero but leaves it at only 12.2%. This is not a successful classifier. It is evidence that loss weighting alone is insufficient under the fixed architecture, training budget and threshold.

Separate training from decision policy

Choose class weights on training or validation, never test. Compare resampling and focal-style losses under equal budgets. Then select a score threshold on validation using precision, recall, review capacity and error cost. A threshold change can increase recall without retraining, but usually reduces precision.

Keep validation and test distributions representative of the intended operating population unless the report explicitly reweights them. Balancing the test set without disclosure changes the meaning of aggregate metrics.

Improve the data before the loss

Check whether minority labels are reliable and whether sources are duplicated. Collect varied minority examples rather than many augmentations of a few sources. Group split before oversampling so copies do not cross partitions. Report performance by source and image-quality slices.

For a safety-critical minority class, define abstention and human review. A model with low recall should fail its release gate even if aggregate accuracy looks acceptable.

The Data Science course links imbalance methods to thresholds, capacity and explicit acceptance criteria.

Exercise

Build a validation precision-recall table for several class weights and thresholds. Add review capacity and false-negative cost. Freeze one policy before the final test and retain failed candidates in the report.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate image models beyond aggregate accuracy.
• Continue with Inspect a model failure without overstating saliency maps.

Reference: scikit-learn precision-recall metrics.