Simpson's paradox in channel conversion rates

An overall conversion ranking can reverse the ranking within every segment when channels contain different segment mixes. Inspect counts and denominators before concluding that one channel performs better for comparable prospects.

The reversal is an arithmetic property of weighted rates. Choosing the appropriate comparison still requires a clear target population and a causal understanding of the segment variable.

Inspect the authored counts

The analyst statistics lab provides the following synthetic channel table:

B has a higher observed rate in both segments. Yet A receives mostly easy prospects, while B receives mostly hard prospects.

import csv
from build_and_verify import ROOT

with (ROOT / 'channel_segments.csv').open(encoding='utf-8', newline='') as handle:
    rows = list(csv.DictReader(handle))
rates = {(r['channel'], r['segment']): int(r['converted'])/int(r['eligible']) for r in rows}
overall = {}
for channel in ['A', 'B']:
    selected = [r for r in rows if r['channel'] == channel]
    overall[channel] = sum(int(r['converted']) for r in selected) / sum(int(r['eligible']) for r in selected)
assert rates['B', 'easy'] > rates['A', 'easy']
assert rates['B', 'hard'] > rates['A', 'hard']
assert overall == {'A': .82, 'B': .35}
standardized = {channel: .5*rates[channel, 'easy'] + .5*rates[channel, 'hard'] for channel in ['A', 'B']}
assert standardized['A'] == .5
assert abs(standardized['B'] - .575) < 1e-12
print({'observed_mix_rates': overall, 'equal_segment_mix_rates': standardized})

At the observed mixes, A converts 82% and B 35%. At an explicitly hypothetical 50/50 segment mix, the standardized rates are 50% and 57.5%, respectively.

Explain which question each rate answers

The overall rates describe each channel's actual observed prospect mix. The standardized rates compare the observed segment-specific rates under a common chosen composition.

Neither rate should be called universally correct without naming the question. A channel manager responsible for the actual delivered portfolio may care about the overall result. A comparison of performance on a common mix may require standardization.

The 50/50 weights are an illustrative target, not a discovered fact about the business. Use the intended target population's weights when they are available and justified.

Do not average rates without denominators

An unweighted mean of segment rates imposes equal segment weights. It does not recover the observed overall conversion rate unless the denominators happen to be equal.

Keep converted and eligible counts in the reporting dataset. A percentage-only export makes it harder to reconstruct the aggregate and assess how much evidence supports each cell.

Avoid automatic adjustment for every available segment

A segment measured before channel assignment may play a different causal role from one created by the channel experience. Conditioning on a mediator or selection variable can answer a different question or introduce bias.

Berkeley's discussion of experiments and confounding includes Simpson-style reversals and the importance of design. The arithmetic here is explicit; a real causal interpretation needs more than the table.

For this fixture, the within-segment comparison is descriptive. The counts do not establish randomized channel assignment or rule out additional differences between prospects.

Turn the finding into a review question

Before reallocating budget, investigate how prospects reach each channel, whether segment definitions are stable and whether the desired target mix differs from the observed mix. Report uncertainty and sample sizes, especially for the ten-person A-hard cell.

Exercise: standardize both channels to 20% easy and 80% hard prospects. Calculate the rates, compare them with the 50/50 result and explain why the channel ranking and the magnitude of the gap are separate properties.

NeuraPath's Data Analytics with Generative AI course connects conversion dashboards with denominator and population design. A useful comparison makes the mix visible before turning a ranking into a recommendation.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Correlation versus causation in a marketing dashboard.
• Continue with Sampling bias in a customer satisfaction survey.