One-hot encoding with unseen categories at inference

A one-hot encoder learns its category vocabulary during fitting. Production will eventually receive a value outside that vocabulary. The system must reject, map or tolerate it by explicit contract.

Observe both behaviours

Our training column contains red, blue, red. The fitted order is blue, red. Test rows contain unseen green and known blue.

With handle_unknown="ignore", green becomes [0,0]; blue becomes [1,0]. The unknown is not assigned to a learned “other” category. It has no active category indicator, so an intercept-only contribution remains for that block.

With strict error mode, transforming green raises ValueError. That can be the safer policy when an unknown value indicates upstream schema failure.

The OneHotEncoder documentation defines unknown-category handling and learned category order.

from feature_cases import onehot_case

r = onehot_case()
assert r['learned_categories'] == ['blue', 'red']
assert r['transformed'][0] == [0, 0]
assert r['transformed'][1] == [1, 0]
assert r['strict_mode_error'] == 'ValueError'
print(r)

Run it in the feature-engineering lab. The exact output makes the serving semantics visible.

Choose reject, ignore or group from the feature meaning

Reject values when the allowed vocabulary is closed and unknowns indicate invalid data. Ignore can keep a service available for genuinely open categories, but monitor unknown rates and downstream performance. An explicit OTHER bucket must be learned through a transformation rule; handle_unknown="ignore" does not create it.

Rare-category grouping can reduce dimensions and stabilize estimates. Fit the grouping threshold on training data only, save the map and apply it unchanged at serving. Distinguish missing from unseen if they have different meanings.

Package the encoder with the model. Do not rebuild a one-hot matrix from incoming categories, because column order and dimension will drift. Test known, unknown, missing and differently cased values through the complete artifact.

An unknown-rate spike may reflect a new product, spelling change, upstream bug or population shift. The response differs, so log raw safe category status and route alerts to the feature owner.

Exercise: add a training-time rare-category grouper and explicit OTHER value. Compare it with all-zero ignore behaviour in a linear model. Write a schema test for case normalization and an alert for unknown share above a declared threshold.

NeuraPath's Data Science course connects encoding choices to serving contracts. Unknown categories require a documented meaning, not only code that avoids crashing.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Target encoding without leaking the target.
• Continue with Feature scaling: fit on training data only.