Target encoding without leaking the target

Target encoding replaces a category with an outcome statistic such as its mean target rate. The feature can compactly represent high-cardinality categories. It can also copy the answer into the input when a row contributes to its own encoding or validation targets enter the category map.

Compare proper and contaminated maps

Our synthetic fixture has 500 rows, 100 categories and category-specific outcome rates. The first 400 rows are training data and the final 100 form validation. Encodings use additive smoothing of five observations toward the 30.5% training rate.

A proper map uses only the 400 training targets. Validation log loss is 0.5766. A contaminated map uses targets from all 500 rows, including validation, and reports the better-looking loss 0.4893. That improvement is leakage, not evidence of better generalization.

For model training, we create five out-of-fold encodings. Each training row is encoded from the other four folds. All 400 rows receive a value, and the fit/holdout parent overlap count is zero. The resulting values range from 0.1173 to 0.6139.

from feature_cases import target_encoding_case

r = target_encoding_case()
assert r['train_rows'] == 400 and r['validation_rows'] == 100
assert r['fold_parent_overlap'] == 0
assert r['out_of_fold_rows'] == 400
assert r['leaky_validation_log_loss'] < r['proper_validation_log_loss']
print(r)

Run the case in the feature-engineering lab. The last assertion documents the expected leakage symptom; it does not endorse the lower score.

Use two maps for two roles

During cross-validation, fit an encoder inside each training fold and transform only its validation fold. When fitting the final development model, create out-of-fold encodings for its training rows, then fit a final category map on all permitted development rows for future inference.

Unknown categories need a fallback such as the training prior. Rare categories need smoothing or grouping. Include counts alongside means during analysis so a category with one positive row does not look equivalent to one with thousands of observations.

Entity and time boundaries still apply. A category map built from future outcomes leaks even if the current row is excluded. For temporal deployment, use only outcomes mature before each prediction time. Group folds when repeated customers or items would otherwise cross boundaries.

Libraries can automate these patterns, but verify them with row identities and a mutation test. Flip one holdout target and confirm its encoding does not change.

Exercise: implement leave-one-out, five-fold and chronological target encodings on the same fixture. Add categories unseen in validation and compare smoothing values. Write tests that mutate held-out targets and detect contamination.

NeuraPath's Data Science course teaches target encoding as a supervised model component. Its fold and time boundaries must be as strict as the classifier's.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with One-hot encoding with unseen categories at inference.
• Then apply it in Feature scaling: fit on training data only.