Detect near-duplicate images across train and test sets

An image test set can look independent while containing a resized crop, recompressed copy or neighboring frame from training. The model may recognize source-specific pixels rather than generalize to a new subject. Split by source group first, then run exact and near-duplicate screening as a second line of defense.

Inject and recover two leaks

The local image lab starts from clean partitions, then replaces one test image with an exact training image and another with a slightly perturbed training image.

from deep_learning_cases import duplicate_case

result = duplicate_case()
assert result["exact_cross_split_matches"] == 1
assert result["near_pixel_matches_rmse_le_0_02"] == 2
print(result["exact_cross_split_matches"], result["near_pixel_matches_rmse_le_0_02"])

An SHA-256 hash finds the exact byte match. Minimum cross-split pixel RMSE at or below 0.02 finds both the exact and perturbed copies. The smallest distance is zero.

The pixel rule works here because every image is aligned, grayscale and 8×8. Real pipelines need similarity methods that tolerate resize, crop, color change and compression. Perceptual hashes or image embeddings can retrieve candidates, but their thresholds create false positives and negatives.

Preserve provenance

The safest split key is usually a patient, video, specimen, product session or original file identifier. Keep all derivatives with that source. If filenames lost provenance, reconstruct groups from capture metadata and similarity clusters before assigning partitions.

For every candidate pair, save source IDs, similarity score, partition and review decision. An image pair can be visually similar without sharing provenance, while two views of the same subject can differ strongly. Human review and business context remain necessary.

Check labels as well as pixels

Exact images with different labels reveal annotation conflict. Near duplicates with the same label inflate test performance. Near duplicates with different labels can signal ambiguous definitions or legitimate temporal change. Report each category separately.

Run the audit before training and again after data refreshes. Fail the build on exact cross-split duplicates unless an explicit reviewed exception exists. For similarity thresholds, sample below and above the cutoff so the chosen operating point is visible.

The Data Science course connects duplicate audits to group-aware splits and evaluation credibility.

Exercise

Create resized, cropped, compressed and brightness-shifted copies. Compare byte hashes, a perceptual hash and embedding distance. Label candidate pairs and choose a screening threshold without consulting model accuracy.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Transfer learning: freeze, fine-tune and compare baselines.
• Continue with Evaluate image models beyond aggregate accuracy.

Reference: scikit-learn group-aware cross-validation guidance.