Multiple comparisons: why twenty tests need a plan

When you test many hypotheses and highlight any result below 0.05, the chance of at least one false positive can be much larger than 5%. Define the family of claims and the error criterion before selecting a correction or reporting only the most favorable result.

Twenty valid individual tests do not automatically form one valid 5% family-level decision.

Calculate the independent all-null example

If twenty independent tests each have a 5% false-positive probability under their true nulls, the probability of at least one false positive is 1−0.95²⁰, approximately 64.15%.

import numpy as np

tests, alpha = 20, .05
theoretical = 1-(1-alpha)**tests
rng = np.random.default_rng(153)
pvalues = rng.uniform(size=(10000, tests))
unadjusted_family_error = float((pvalues < alpha).any(axis=1).mean())
bonferroni_family_error = float((pvalues < alpha/tests).any(axis=1).mean())
assert abs(theoretical-.6415140775914581) < 1e-12
assert .62 < unadjusted_family_error < .67
assert .035 < bonferroni_family_error < .065
assert alpha/tests == .0025
print({'theoretical_unadjusted': theoretical,
       'simulated_unadjusted': unadjusted_family_error,
       'simulated_bonferroni': bonferroni_family_error})

The simulation uses independent uniform p-values to represent valid tests when all nulls are true. It runs in the analyst statistics lab. It is not a measured false-positive rate for a particular company's experiments.

Understand the Bonferroni guarantee

For a family of twenty tests and family-wise alpha 0.05, Bonferroni uses an individual threshold of 0.0025. With valid individual tests, its family-wise error bound does not require independence, although the simple 64.15% calculation above does.

NIST's Bonferroni method guide explains the simultaneous-error principle. The method can be conservative, especially with related tests, so the choice should follow the intended claims and design.

Other procedures address different error criteria. Controlling false discovery rate is not the same promise as controlling the chance of any false positive in a family. State which criterion the analysis uses.

Define the family from the decision

The family may include several outcomes, treatment variants, segments or planned contrasts. It should not be redefined after seeing which subset gives the desired answer.

If the business has one primary outcome and several guardrails, document how each contributes to the decision. A dashboard containing twenty exploratory metrics need not imply twenty identical confirmatory claims, but the reporting must preserve that distinction.

Keep exploratory findings useful

Exploration can generate hypotheses and reveal unexpected behavior. Label findings as exploratory, report the search scope and seek confirmation with suitable data or a planned follow-up.

Do not present a subgroup discovered after trying many segment definitions as if it had been the sole prespecified test. The selection process affects how persuasive the result is.

Similarly, multiple-comparison adjustment does not repair biased sampling, invalid standard errors or incorrect outcome definitions. The individual tests must first be appropriate.

Record all relevant results

Retain the tested hypotheses, raw p-values, adjustment method, adjusted results and effect estimates with intervals where appropriate. A table of only significant outcomes hides the denominator of the analytical search.

A nonselected result may still be informative about uncertainty or practical magnitude. Avoid reducing the entire analysis to a list of winners and losers.

Exercise: simulate ten highly correlated tests by reusing a shared random component. Compare the observed family error with the independent formula, then explain why Bonferroni's bound and the exact independence calculation are different statements.

NeuraPath's Data Analytics with Generative AI course connects hypothesis testing with an explicit analysis plan. A credible set of claims accounts for how many opportunities the analysis had to find a favorable result.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Effect size versus statistical significance for analysts.
• Continue with A/B testing randomization checks before reading the uplift.