Imputation strategies under different missingness patterns

Mean, median and model-based imputation answer different assumptions. Their suitability depends on distribution, missingness mechanism and downstream objective. A single overall metric can hide severe error for the missing subgroup.

Retain hidden values for a synthetic audit

We generate a skewed positive feature. Its training mean is 10.4405 and median 6.9921. In one test pattern, 43 of 150 values are hidden at random. Their actual mean happens to be 12.3777. Mean-imputation RMSE is 11.7029; median RMSE is 12.7361.

In the second pattern, every value above the test 75th percentile is hidden, producing 38 missing rows with true mean 25.0601. Mean-imputation RMSE rises to 20.2303 and median RMSE to 22.8470. Both constants systematically understate the hidden high-value group.

from feature_cases import imputation_case

r = imputation_case()
assert r['training_mean'] > r['training_median']
assert r['patterns']['mcar']['missing_rows'] == 43
assert r['patterns']['high_values_missing']['missing_rows'] == 38
assert r['patterns']['high_values_missing']['true_missing_mean'] > 25
print(r)

Run the construction in the feature-engineering lab. True hidden values are visible only because this is synthetic; real missing values cannot directly reveal their own imputation error.

Evaluate the downstream task and the mechanism

Mean imputation is sensitive to skew and outliers. Median is robust but still collapses missing rows to one value. Group-wise and model-based imputers can use more structure, while introducing more fitted components and leakage risk. Multiple imputation targets parameter uncertainty in inferential settings and should not be reduced to a buzzword.

Fit every imputer on training data inside cross-validation. Add missing indicators when justified. Compare downstream metrics, calibration and missing-subgroup behaviour, not only reconstruction on observed values artificially masked at random.

Investigate why data are absent and whether serving will reproduce the training mechanism. If values become missing exactly when they are large, no constant imputer can recover their distribution without additional information or assumptions.

Exercise: add a correlated auxiliary feature and compare median, group median and iterative prediction under both patterns. Fit inside folds, report missing-subgroup error and state which method depends most strongly on a stable relationship.

NeuraPath's Data Science course teaches imputation through missingness scenarios and held-out evaluation. The strategy follows the data process, not a universal default.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Missing-value indicators: when missingness carries information.
• Continue with Create time-based features without looking into the future.