Covariance matrices: detect redundant features

A covariance matrix summarizes how centered numeric features vary together. Exact linear redundancy creates a direction with no variation and makes the matrix singular. That is a property of the feature representation, not an error to hide by repeatedly trying a matrix inverse.

Our original four-row fixture has x=(1,2,3,4) and a second feature z=2x. The second column contains no additional information beyond the first. Using sample covariance with divisor n-1 gives:

[[5/3, 10/3], [10/3, 20/3]].

The diagonal entries are feature variances; the off-diagonal entries are covariances. Their magnitudes depend on the feature units.

Find the direction with zero variation

The vector (-2,1) represents the combination -2*x+z, which is zero in every row. Multiplying the covariance matrix by this vector also returns zero. Its eigenvalues are zero and 25/3, so the matrix has rank one rather than two.

NumPy's symmetric eigenvalue documentation describes the decomposition used for symmetric matrices. The included example is small enough to check the null direction without relying solely on an eigensolver.

import numpy as np
from model_geometry import redundancy_case

r = redundancy_case()
cov = np.array(r['covariance'])
assert np.allclose(cov,[[5/3,10/3],[10/3,20/3]])
assert r['rank']==1
assert np.allclose(cov@np.array([-2.,1.]),[0.,0.])
assert np.allclose(r['eigenvalues'],[0.,25/3])
x = np.arange(1.,5.)
assert np.allclose(2*x, 0*x+1*(2*x))
assert np.allclose(2*x, 4*x-1*(2*x))
print(r)

Run from the mathematics lab. The last two assertions show different coefficient pairs producing identical predictions: (0,1) and (4,-1) both represent 2x when the columns are x and 2x.

Separate coefficient uniqueness from predictive behavior

When several coefficient combinations produce the same fitted values, interpreting one coefficient as a uniquely estimated effect is misleading. A solver can still return a particular solution, for example through a pseudoinverse or penalty. That computational choice does not create new information in the data.

Likewise, a well-behaved prediction on observed feature combinations does not prove stable behavior when future columns break the training relationship. If z stops equalling 2x because of a changed upstream definition, the fitted model's behavior can depend on which coefficient solution was chosen.

Deal with near redundancy carefully

Real datasets often have approximate rather than exact dependence. Tiny noise can turn a zero eigenvalue into a small positive value without making coefficient estimation practically stable. Numerical rank depends on a tolerance, and feature scaling affects the conditioning of the matrix.

Inspect definitions first: duplicated currencies, totals alongside all their components, or two versions of the same measurement may explain the issue more clearly than a heatmap alone. Pairwise correlation also cannot detect every multicolumn dependency; one column can be a combination of several others.

Potential responses include removing a justified duplicate, changing the representation, dimensionality reduction or regularization. Choose according to the prediction and interpretation requirements, then evaluate the resulting pipeline. Do not delete features solely because a fixed correlation threshold was crossed.

Exercise: replace z with 2*x + [0,.01,-.01,0]. Compare the eigenvalues and condition number with the exact case. Record how your numerical tolerance affects a rank statement and explain why a technically invertible matrix can remain troublesome.

NeuraPath's Data Science course connects linear algebra with feature engineering. The deliverable is a diagnosis of the redundant information and a justified modelling response, not merely a successful inverse calculation.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Cosine similarity versus Euclidean distance after scaling.
• Continue with Log transformations: interpret predictions on the original scale.