Sampling bias in a customer satisfaction survey

A satisfaction mean describes the people who supplied usable responses unless the design and adjustment support a broader target population. Highly engaged customers may respond more often than quiet or dissatisfied customers, so a large response count can still represent the wrong mix.

Define the target population, invitation process and response coverage before interpreting the headline score.

Reproduce a composition-driven difference

The analyst statistics lab defines a synthetic population of 1,000 customers: 200 engaged and 800 other customers. Their supplied segment means are 5 and 2. The 100 respondents contain 80 engaged and only 20 other customers.

import csv
from build_and_verify import ROOT

with (ROOT / 'survey_segments.csv').open(encoding='utf-8', newline='') as handle:
    rows = list(csv.DictReader(handle))
population_n = sum(int(r['population_n']) for r in rows)
respondents = sum(int(r['respondents']) for r in rows)
respondent_mean = sum(int(r['respondents'])*float(r['mean_score']) for r in rows)/respondents
target_mean = sum(int(r['population_n'])*float(r['mean_score']) for r in rows)/population_n
weights = {r['segment']: int(r['population_n'])/int(r['respondents']) for r in rows}
weighted_mean = sum(weights[r['segment']]*int(r['respondents'])*float(r['mean_score']) for r in rows) / sum(weights[r['segment']]*int(r['respondents']) for r in rows)
assert respondent_mean == 4.4
assert target_mean == weighted_mean == 2.6
assert weights == {'engaged': 2.5, 'other': 40.0}
print({'respondents': respondents, 'respondent_mean': respondent_mean,
       'target_mean_under_supplied_segment_means': target_mean, 'weights': weights})

The unweighted respondent mean is 4.4, while the target-composition mean is 2.6. This is a deliberately constructed teaching example, not a report of NeuraPath or competitor satisfaction.

State why the weighting works here

The example supplies the target segment counts and assumes the observed segment means apply to all customers within each segment. Under those conditions, weighting restores the target composition.

In real data, nonrespondents within a segment may differ from respondents. Matching segment proportions does not automatically repair that difference. Missing coverage, inaccurate segment labels and extreme weights can also complicate the result.

The Census Bureau's weighting explanation describes adjustment for differential representation. The fixture's simple weights illustrate one mechanism; they are not a substitute for a full survey design.

Inspect response rates and the invitation frame

If all 1,000 customers were invited in this hypothetical design, the overall response rate is 10%. Engaged response is 80/200 = 40%, while other-customer response is 20/800 = 2.5%.

The difference signals a representation problem to investigate. Response rate alone does not determine the exact bias: bias also depends on how respondents and nonrespondents differ on the measured outcome.

The Census Bureau's response-rate definitions explain response rates as data-quality measures and discuss nonresponse concerns.

Do not treat a narrow interval as a bias correction

An ordinary confidence interval around the respondent mean reflects uncertainty under its sampling assumptions. It does not automatically include the error caused by selecting a different population.

Collecting more responses through the same highly selective channel can estimate the respondent population more precisely while leaving target-population bias unresolved.

Improve the invitation frame, contact strategy and accessibility of the survey where appropriate. Use auxiliary information and follow-up studies to assess nonresponse rather than assuming a weighting formula has solved it.

Report the population honestly

A useful report states who was eligible, who was invited, who responded, the scale, missing-answer rules, any weights and the assumptions behind generalization. Keep weighted and unweighted results distinguishable.

Avoid saying customers rate us 4.4 when the supported statement is responding customers averaged 4.4 under the specified survey. The wording should match the evidence.

Exercise: change the unobserved other-customer mean while leaving respondents' observed mean unchanged. Show why the same response data can be compatible with different target means unless the within-segment representativeness assumption is justified.

NeuraPath's Data Analytics with Generative AI course connects survey analysis with population reasoning. A credible satisfaction report explains whose experience the number represents and which customers remain underrepresented.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Simpson's paradox in channel conversion rates.
• Continue with Bootstrap a median delivery-time interval.