Explain an inconclusive experiment to a business stakeholder

Lead with the estimated effect, the uncertainty range and the decision-relevant possibilities still compatible with the data. An inconclusive result does not prove no effect, and it does not justify presenting the positive point estimate as an established win.

Explain what the experiment ruled out, what it did not and what additional action would be justified by a new plan.

Calculate the worked result

The analyst statistics lab supplies 400 conversions among 10,000 control units and 450 among 10,000 treatment units. All numbers are synthetic.

import csv
import numpy as np
from scipy import stats
from build_and_verify import ROOT

with (ROOT / 'ab_counts.csv').open(encoding='utf-8', newline='') as handle:
    rows = {row['group']: row for row in csv.DictReader(handle)}
n0, n1 = int(rows['control']['assigned']), int(rows['treatment']['assigned'])
x0, x1 = int(rows['control']['converted']), int(rows['treatment']['converted'])
p0, p1 = x0/n0, x1/n1
difference = p1-p0
se_interval = np.sqrt(p0*(1-p0)/n0+p1*(1-p1)/n1)
low, high = difference + np.array([-1, 1])*stats.norm.ppf(.975)*se_interval
pooled = (x0+x1)/(n0+n1)
z = difference/np.sqrt(pooled*(1-pooled)*(1/n0+1/n1))
pvalue = 2*stats.norm.sf(abs(z))
assert np.isclose(difference, .005)
assert np.isclose(p1/p0-1, .125)
assert low < 0 < .01 < high
assert pvalue > .05
print({'control_rate': p0, 'treatment_rate': p1, 'difference_percentage_points': 100*difference,
       'relative_lift_percent': 100*(p1/p0-1), 'ci95_percentage_points': [100*low, 100*high],
       'two_sided_p': float(pvalue)})

The observed difference is +0.5 percentage points, or +12.5% relative to the 4% control rate. The approximate 95% interval is −0.059 to +1.059 percentage points.

This uses an unpooled normal interval and a pooled two-proportion z test under independent-unit, complete-outcome assumptions. They are declared large-sample approximations, not a universal method for every experiment. SciPy's normal distribution reference supports the numerical tail and quantile calculations.

Use decision language that preserves uncertainty

A suitable explanation is: the observed conversion increase was half a percentage point, but the interval still includes zero and the scenario's meaningful improvement of one percentage point. The current evidence does not distinguish a negligible effect from an improvement large enough to matter under that threshold.

Do not say the treatment definitely has no effect because p exceeds 0.05. Do not say conversion improved by 12.5% without making clear that this is the observed estimate with uncertainty.

Check whether the result is interpretable at all

Before discussing inconclusiveness, verify assignment quality, source completeness, outcome maturation and the analysis plan. A corrupted experiment is not merely a low-powered valid experiment.

The aggregate fixture cannot establish that operational guardrails passed. The worked analysis plan identifies those additional evidence requirements.

If instrumentation failed, prioritize repair and a defensible new evaluation rather than treating the current interval as a complete decision input.

Discuss options without moving the goalposts

The next action can depend on implementation cost, reversibility, plausible harm, guardrails and the value of more information. Options might include retaining the current experience, redesigning the treatment or planning a new study around the remaining uncertainty.

Do not extend a completed fixed-horizon experiment solely until its p-value crosses 0.05. A new data-collection or sequential-analysis plan should account for the decisions already made.

Likewise, do not search many segments and present a favorable subgroup as if it were the original primary result. Exploratory findings can motivate a follow-up, with their selection history disclosed.

Preserve the learning

An inconclusive result still documents the observed effect, uncertainty, execution quality and feasibility of detecting the meaningful threshold. Those facts can improve future planning even when they do not authorize a rollout.

Exercise: write a four-sentence stakeholder update containing the absolute effect, interval, meaningful threshold and proposed next step. Remove any sentence that equates nonsignificance with proof of no effect or presents the point estimate as certain.

NeuraPath's Data Analytics with Generative AI course connects experiment analysis with clear business communication. A useful update makes uncertainty actionable without hiding it.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Build a statistical analysis plan before opening the results.
• Return to the cluster foundation in Mean versus median in a skewed order-value report.