Sequential peeking: why stopping at significance misleads

Repeatedly checking a fixed-horizon p-value and stopping as soon as it falls below 0.05 changes the testing procedure. The ordinary 5% false-positive interpretation does not automatically survive that stopping rule.

Monitoring an experiment is necessary for operations and guardrails. The inferential method must be designed for the decisions made during monitoring.

Simulate the stopping rule directly

This teaching example tests a normal mean against zero with known standard deviation one. The null is true in every simulated experiment. Each experiment has up to 200 independent observations, with looks after 20, 40 and so on through 200.

import numpy as np
from scipy import stats

rng = np.random.default_rng(156)
observations = rng.normal(0, 1, size=(20000, 200))
looks = np.arange(20, 201, 20)
z = observations.cumsum(axis=1)[:, looks-1] / np.sqrt(looks)
pvalues = 2*stats.norm.sf(abs(z))
fixed_horizon_error = float((pvalues[:, -1] < .05).mean())
stop_when_significant_error = float((pvalues < .05).any(axis=1).mean())
assert .04 < fixed_horizon_error < .065
assert stop_when_significant_error > fixed_horizon_error + .1
print({'experiments': 20000, 'looks': len(looks),
       'fixed_horizon_false_positive_rate': fixed_horizon_error,
       'stop_on_significance_false_positive_rate': stop_when_significant_error})

The recorded rates are 5.185% at the fixed horizon and 19.56% for stopping at any significant look. These are simulation results for this specific model and schedule, not universal peeking rates.

The code runs in the analyst statistics lab. Johari, Pekelis and Walsh's Always Valid Inference paper studies inference designed for continuous monitoring in A/B testing.

Explain why the opportunities accumulate

The same evolving experiment can cross a threshold by chance at one look and move back later. Selecting the first favorable crossing changes which outcomes become reported conclusions.

The looks are correlated because later samples contain earlier observations. Therefore, the twenty-independent-tests formula is not the calculation for this example. Simulating the actual dependence and stopping rule makes the distinction visible.

Choose a design that matches the intended decisions

A fixed-horizon plan can specify the sample size and analysis time in advance, while operational monitoring follows separate rules. Group-sequential designs can allocate error across planned looks. Always-valid approaches can support other monitoring policies under their own assumptions.

These methods are not interchangeable labels. Specify the test, boundary or confidence procedure, outcome, unit and stopping policy before treating interim results as decision evidence.

The example demonstrates the defect in an unplanned rule; it does not implement a production sequential-testing system.

Keep safety and data-quality monitoring distinct

An experiment may need to stop for severe operational harm or broken instrumentation regardless of a primary-outcome p-value. Record that stop reason and avoid presenting the resulting truncated data as if the original fixed-horizon analysis completed unchanged.

Likewise, extending an experiment only because its p-value is just above 0.05 is another outcome-dependent decision. A longer run is not automatically more valid if its duration was selected to obtain a preferred result.

Report the path that produced the conclusion

Record planned and actual sample sizes, analysis looks, stopping reason, primary outcome and any deviations. A final p-value without the monitoring history can conceal how much selection occurred.

Also keep practical effect size and uncertainty visible. Reaching a significance boundary does not establish that an effect is large enough to justify deployment or that guardrails are acceptable.

Exercise: rerun the simulation with only two planned looks and then with a look after every observation from 20 onward. Compare the stop-on-significance rates and explain why the monitoring schedule is part of the statistical procedure.

NeuraPath's Data Analytics with Generative AI course connects experimentation with valid stopping rules. A trustworthy result accounts for how the analyst decided when to look and when to stop.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Calculate sample size from a meaningful business effect.
• Continue with Missing data mechanisms with an analyst's decision checklist.