PCA for compression without leaking test information

PCA does not use the target, but it learns means and directions from data. Fitting scaling or PCA on test rows lets their distribution shape the representation before evaluation. Unsupervised preprocessing can still leak.

Create a visible distribution shift

Our construction has 240 training rows and 120 test rows with six numeric features. Training feature zero has mean -0.1240. Test feature zero is shifted and has mean 3.0029.

A scaler fitted on training alone stores -0.1240 for that feature. A leaky scaler fitted on all 360 rows stores 0.9183, which incorporates the future test distribution. The first two proper PCA components explain 19.87% and 19.45% of standardized training variance. The leaky components report 19.46% and 18.45% on a different full-data coordinate system.

The leaky test reconstruction MSE appears smaller, 0.7488 versus 1.7226 for the proper pipeline. Those errors use different scalers and bases, so ranking them as evidence of better generalization would be invalid. The leaky pipeline has adapted to test shift.

The PCA documentation states that input is centred before singular-value decomposition. Scaler and PCA statistics belong inside the training pipeline.

import numpy as np
from unsupervised_cases import pca_case

r = pca_case()
assert np.isclose(r['proper_scaler_feature0_mean'], r['train_feature0_mean'])
assert not np.isclose(r['leaky_scaler_feature0_mean'], r['train_feature0_mean'])
assert r['test_feature0_mean'] > 3
print(r)

Run this in the unsupervised lab. The shift is synthetic and intentionally obvious.

Fit the whole transformation inside resampling

For cross-validation, each fold fits imputation, scaling and PCA on its fit partition, then transforms its validation partition. A pipeline enforces that order. After selecting the component count, refit on permitted development data and evaluate once on untouched test rows.

Select components using a declared rule: explained-variance target, validation performance, reconstruction requirement or downstream cost. Explained variance is not predictive value. Low-variance directions can contain the target signal, and high-variance directions can reflect noise or batch effects.

Persist the scaler, PCA object and feature order with the model. Serving must apply the same transformations. Monitor input drift and reconstruction behaviour, but avoid silently refitting components because that changes every downstream coordinate.

Exercise: place PCA before a classifier in a cross-validation pipeline and compare it with a full-data PCA fitted before splitting. Add a feature-order permutation at serving time and write a schema check that fails before prediction.

NeuraPath's Data Science course connects dimensionality reduction to split discipline and artifact packaging. “Unsupervised” describes the objective, not permission to inspect the test distribution.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Cluster stability under resampling.
• Continue with t-SNE and UMAP: why a picture does not prove clusters.