Calculate sample size from a meaningful business effect

Choose a meaningful effect before calculating sample size. For a conversion experiment, specify the baseline probability, absolute difference, allocation ratio, significance level, desired power and unit of randomization.

Sample size is conditional on those assumptions. It is not a promise that an experiment will produce significance or that its true effect equals the planning effect.

Define a concrete planning scenario

Suppose the baseline conversion is 4% and the hypothetical business requires detecting an increase to 5%. That is a one-percentage-point absolute effect and a 25% relative increase, not a 1% relative increase.

Plan equal independent groups, a two-sided alpha of 0.05 and approximately 80% power. The following common normal approximation combines null and alternative standard deviations and ignores the very small opposite-tail contribution when solving the planning equation.

import math
import numpy as np
from scipy import stats

def planned_n(p0, p1, alpha=.05, power=.8):
    if not (0 < p0 < 1 and 0 < p1 < 1 and p0 != p1 and 0 < alpha < 1 and .5 < power < 1):
        raise ValueError('invalid_planning_inputs')
    pooled = (p0+p1)/2
    numerator = (stats.norm.ppf(1-alpha/2)*math.sqrt(2*pooled*(1-pooled))
                 + stats.norm.ppf(power)*math.sqrt(p0*(1-p0)+p1*(1-p1)))
    return math.ceil(numerator**2/(p1-p0)**2)

n = planned_n(.04, .05)
assert n == 6745
assert planned_n(.04, .045) > 3*n
rng = np.random.default_rng(155)
control = rng.binomial(n, .04, size=10000)
treatment = rng.binomial(n, .05, size=10000)
pooled = (control+treatment)/(2*n)
z = (treatment/n-control/n)/np.sqrt(pooled*(1-pooled)*2/n)
simulated_power = float((2*stats.norm.sf(abs(z)) < .05).mean())
assert .77 < simulated_power < .83
print({'n_per_group': n, 'total_n': 2*n, 'simulated_power': simulated_power})

The result is 6,745 units per group, 13,490 total under this approximation. The independent Bernoulli simulation checks that the corresponding pooled-proportion z test has power near the intended level under the stated effect.

The statsmodels sample-size documentation describes this normal-approximation approach and its tail approximation. The code above implements the formula directly with SciPy and runs in the analyst statistics lab.

Treat the baseline as an assumption to examine

Use a representative historical baseline and consider plausible alternatives. A seasonal or selected baseline can produce an unrealistic plan. If the effect of interest is smaller, substantially more information may be required.

Do not choose a larger planning effect merely to fit available traffic and then claim the study can reliably detect a smaller one. Report the tradeoff honestly.

Count the correct units

The calculation assumes independent Bernoulli outcomes per randomized unit. Repeated visits from one user do not supply the same information as independent users. Cluster randomization, unequal allocation, attrition and exposure delays require appropriate adjustments or a different calculation.

Also define the conversion window. Users enrolled near the end of recruitment may need additional follow-up before their outcomes are complete.

Translate sample size into a feasible schedule

Estimate eligible unique traffic, allocation and outcome maturation. Cover relevant business cycles when the design requires them. A simple division by daily page views can understate duration if many views belong to the same users or most traffic is ineligible.

Keep operational guardrails and stopping rules separate from the fixed-horizon significance calculation. If sequential monitoring is planned, use a design and sample-size approach that supports it.

Exercise: compare planning for 4%→4.5%, 4%→5% and 4%→6%. Report absolute and relative effects, required units and the decision rationale for each rather than choosing the shortest experiment automatically.

NeuraPath's Data Analytics with Generative AI course connects experiment planning with practical effect definitions. A useful sample-size calculation makes its assumptions reviewable before data collection begins.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in A/B testing randomization checks before reading the uplift.
• Continue with Sequential peeking: why stopping at significance misleads.