Missing-value indicators: when missingness carries information

Imputation fills a value so a model can compute. A missingness indicator preserves the fact that a value was absent. That fact can be predictive when collection, eligibility or behaviour affects whether the field is observed.

Create an informative missingness mechanism

Our synthetic target prevalence is 30%. The numeric value itself is random noise. Positive rows have a 65% missing probability and negative rows 10%. Training and test missing rates are 25.88% and 27.75%.

Both pipelines use training-median imputation and logistic regression. The first receives only the imputed numeric value. The second adds a binary missingness indicator through SimpleImputer(add_indicator=True).

On 400 untouched test rows, the imputed-only pipeline has AUC 0.5278 and log loss 0.6065. Adding the indicator yields AUC 0.7760 and log loss 0.4667. The gain reflects the authored missingness process, not information recovered from the absent numeric values.

The SimpleImputer documentation defines the optional indicator and its fit-time behaviour.

from feature_cases import missing_indicator_case

r = missing_indicator_case()
base = r['test']['imputed_only']
flag = r['test']['with_indicator']
assert flag['auc'] > base['auc']
assert flag['log_loss'] < base['log_loss']
assert r['test_missing_rate'] > .27
print(r)

Run it in the feature-engineering lab. The generator deliberately makes absence outcome-associated; real mechanisms must be investigated.

Ask why the value is missing

Missingness can mean “not applicable,” “not collected,” “collection failed” or “not yet available.” Combine states only when their semantics match. A missing flag can become a proxy for site, device, staff process or eligibility.

The signal may disappear after a form or pipeline change. Monitor missing rates and indicator coefficients or importance by period. Do not encourage operational teams to withhold data because absence raises a score.

Fit imputation values and indicator columns on training data. Some implementations create indicators only for features missing during fit, so a feature first missing at serving may not receive a flag. Test this case and maintain a schema-level missingness policy.

Exercise: create separate codes for not-applicable and collection-failed states. Change missingness rates in a later period and compare calibration. Define a monitoring alert that identifies process drift without claiming the missing flag causes the outcome.

NeuraPath's Data Science course connects missing-value features to their data-generating process. An indicator is useful when its meaning remains observable and governed.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Feature scaling: fit on training data only.
• Continue with Imputation strategies under different missingness patterns.