ANOVA: what a significant result does not tell you

A significant one-way ANOVA result provides evidence against the hypothesis that all population group means are equal, under the model's assumptions. It does not identify every differing pair, establish that all groups differ or prove that the group assignment caused the outcome.

Plan the comparisons and practical effect interpretation before treating an omnibus p-value as a business conclusion.

Construct three groups with different roles

In this synthetic example, groups A and B are close together while C is much higher. The five observations per group are independent under the teaching model, with a common within-group variance assumption for the classical ANOVA.

import numpy as np
from scipy import stats

a = np.array([9, 10, 11, 10, 10], dtype=float)
b = np.array([10, 11, 12, 11, 11], dtype=float)
c = np.array([19, 20, 21, 20, 20], dtype=float)
omnibus = stats.f_oneway(a, b, c, equal_var=True, nan_policy='raise')
pairwise = stats.tukey_hsd(a, b, c, equal_var=True)
assert [a.mean(), b.mean(), c.mean()] == [10, 11, 20]
assert omnibus.pvalue < .001
assert pairwise.pvalue[0, 1] > .05
assert pairwise.pvalue[0, 2] < .05 and pairwise.pvalue[1, 2] < .05
print({'F': float(omnibus.statistic), 'omnibus_p': float(omnibus.pvalue),
       'Tukey_pairwise_p': pairwise.pvalue.tolist()})

The result demonstrates that a small omnibus p-value does not imply evidence for every pairwise difference. A versus B does not pass the 0.05 threshold in this specified Tukey comparison, while comparisons with C do.

Failure to reject A versus B is not proof of equivalence. Establishing that differences are small enough for a decision requires an appropriate equivalence or interval-based analysis with a meaningful margin.

Understand what the omnibus statistic compares

Classical one-way ANOVA compares variation between group means with variation within groups. NIST's ANOVA table guide explains this variance-ratio construction.

The SciPy f_oneway reference documents the equal-variance and Welch options. The Tukey HSD reference documents the pairwise procedure used above. The code was executed in the analyst statistics lab with SciPy 1.18.0.

Choose follow-up comparisons deliberately

If the decision concerns a small set of planned contrasts, specify them in the analysis plan. If all pairwise comparisons are relevant, use an appropriate multiplicity-aware procedure and report its uncertainty and assumptions.

Running many unadjusted t tests after a significant omnibus result does not automatically control the error rate for every claim you choose to report. The family of intended conclusions matters.

Check the design behind the groups

Repeated measurements on the same people are not independent groups. Several observations from one team or branch may also be clustered. The analysis must reflect the unit that supplies independent information.

For small samples, normal-error assumptions and unusual values deserve careful review. Welch's ANOVA relaxes equal population variances for independent groups; it does not fix dependence, selection bias or every distributional problem.

The fixture's simple values are chosen to illustrate interpretation, not to certify assumptions for a real dataset.

Report magnitude in the original scale

The observed A–B difference is one score unit, while C exceeds A by ten units. Whether those differences matter depends on the scale and a decision threshold. A p-value does not supply that threshold.

Also report sample sizes, group summaries and relevant intervals. A chart of means without spread or design context can exaggerate certainty, especially with only five observations per group.

Exercise: shift all three groups upward by 100 and verify that the ANOVA and pairwise p-values remain unchanged while the group means move. Explain why the test concerns differences relative to variability, not the absolute origin of the score scale.

NeuraPath's Data Analytics with Generative AI course connects multi-group testing with responsible interpretation. A useful ANOVA report states which hypothesis was tested and which follow-up claims the evidence actually supports.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Chi-square tests: check expected counts before interpreting results.
• Continue with Correlation versus causation in a marketing dashboard.