A/B testing randomization checks before reading the uplift

Before interpreting uplift, verify that assignment, exposure and outcome records represent the intended experiment population. Check the configured allocation ratio, stable unit identity, cross-variant contamination and missing telemetry.

A favorable conversion difference is not trustworthy if the data pipeline selectively loses units from one variant.

Test the configured allocation ratio

For a hypothetical 50/50 experiment with 10,000 independently assigned units, expected group counts are 5,000 and 5,000. An observed 6,000/4,000 split warrants investigation.

import numpy as np
from scipy import stats

expected_ratio = np.array([.5, .5])
observed = np.array([6000, 4000])
expected = observed.sum() * expected_ratio
result = stats.chisquare(observed, f_exp=expected)
assert result.statistic == 400
assert result.pvalue < .001
balanced = stats.chisquare([5000, 5000], f_exp=[5000, 5000])
assert balanced.statistic == 0 and balanced.pvalue == 1
print({'mismatch_statistic': float(result.statistic), 'mismatch_p': float(result.pvalue),
       'balanced_example_p': float(balanced.pvalue)})

This sample-ratio mismatch check compares counts with the configured allocation, not conversion outcomes. The 0.001 assertion is an illustrative detection check for an extreme fixture, not a universal operational threshold.

The example runs in the analyst statistics lab. Microsoft's SRM diagnosis article describes how assignment, execution, logging and analysis problems can produce mismatches and why diagnosis precedes effect interpretation.

Use the correct expected ratio

A 60/40 split is not evidence of an error when the experiment deliberately assigns 60% and 40%. Ramping, eligibility windows and multiple allocation stages can also change the appropriate expectation.

Compare each relevant period or stratum with its actual configuration. Do not force a 50/50 expectation onto a design that never promised it.

Repeated monitoring of an SRM alarm also needs a planned monitoring policy. An arbitrary threshold checked continuously does not have the same interpretation as one prespecified check.

Validate identity separately

from collections import defaultdict

assignments = [('U1', 'A'), ('U2', 'B'), ('U3', 'A'), ('U1', 'B')]
variants_by_unit = defaultdict(set)
for unit, variant in assignments:
    variants_by_unit[unit].add(variant)
conflicts = sorted(unit for unit, variants in variants_by_unit.items() if len(variants) > 1)
assert conflicts == ['U1']
assert len(assignments) == 4 and len(variants_by_unit) == 3
print({'units_assigned_to_multiple_variants': conflicts})

Equal row counts could coexist with cross-variant identity problems. Count unique assignment units and define whether that unit is a user, account, household or another entity. Repeated visits are not automatically new randomized units.

Compare assignment and analysis populations

An experiment may assign a user successfully but fail to log exposure or outcomes. Inspect missingness by variant and trace joins from the assignment table to the final analysis table.

Filtering to people who clicked a treatment-specific feature can create a selected population affected by the treatment. The comparison then differs from the originally randomized assignment population.

Define eligibility and any triggered analysis carefully, with equivalent measurement across variants. Preserve counts at each stage so the first divergence can be located.

Interpret passing checks modestly

No detected SRM does not prove randomization was implemented correctly or that every metric is unbiased. Balanced counts can hide incorrect bucketing, differential outcomes logging or interference between units.

Review implementation and data-quality evidence together. Baseline covariate checks can help diagnose anomalies, but chance imbalances also occur under valid randomization; they should not become a reason to repeatedly rerandomize until every p-value looks favorable without a planned design.

Exercise: construct an assignment table with balanced variants, then drop outcome records only for high-activity treatment users. Show how assignment SRM can pass while the analysis population becomes biased, and identify the stage-count check that reveals the loss.

NeuraPath's Data Analytics with Generative AI course connects experimentation with data engineering checks. A useful uplift report first establishes which units survived from assignment to analysis.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Multiple comparisons: why twenty tests need a plan.
• Continue with Calculate sample size from a meaningful business effect.