Compare group error rates with sample-size context

Group error rates can expose concentrated failures that aggregate metrics hide. A rate without its denominator, positive support and threshold can also exaggerate noise or invite an unsupported fairness conclusion.

Keep the full table together

We apply one validation-selected threshold, 0.25148, to the 1,500-row synthetic test period and split results by an authored new versus returning segment.

Recall is similar in this sample, while precision and average precision differ. The groups also have different sizes and prevalence. The table is descriptive; it neither certifies fairness nor identifies why the differences occurred.

from imbalance_cases import group_case

r = group_case()
groups = {row['segment']: row for row in r['groups']}
assert groups['new']['n'] == 450 and groups['new']['positives'] == 34
assert groups['returning']['n'] == 1050 and groups['returning']['positives'] == 61
assert sum(row['tp'] + row['fn'] for row in groups.values()) == 95
print(r)

Run the calculation in the imbalanced-model lab. The group label and outcomes are synthetic and carry no claim about a protected population.

Define the comparison before testing it

Choose groups that match the decision and legal or ethical review context. Define whether identities are self-reported, inferred, missing or mutable. Intersections can matter even when single dimensions look similar, but slice counts shrink quickly.

Report confusion counts, rates, denominators and uncertainty. Repeated accounts require cluster-aware intervals. Multiple group and metric comparisons increase the chance of apparently large differences, so predeclare priority analyses and treat exploratory findings as hypotheses.

Different prevalence changes precision even when conditional error rates are identical. Different feature quality or label processes can also create gaps. Decomposing those mechanisms needs additional evidence; the metric table alone does not establish discrimination, causation or the correct remedy.

The operating threshold matters. A global threshold may produce different workloads and rates by group. Group-specific thresholds can change those trade-offs but raise policy, calibration and governance questions. Do not choose them from the final test set.

Exercise: bootstrap accounts to estimate paired uncertainty for the recall difference, then add an intersection with a second synthetic segment. Report cells with inadequate positive support and write a rule for suppressing unstable estimates without hiding failures.

NeuraPath's Data Science course treats subgroup evaluation as a documented analysis, not a single fairness number. Counts, definitions and uncertainty make the comparison reviewable.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Threshold selection with unequal error costs.
• Continue with Abstention: let a classifier decline uncertain cases.