Ridge versus lasso when predictors are correlated

With strongly correlated predictors, several coefficient combinations can produce similar predictions. Ridge and lasso impose different preferences among those combinations. A nonzero lasso coefficient does not, by itself, prove that the selected feature has a uniquely important effect.

An extreme original example makes this visible. Let x=(-2,-1,0,1,2), duplicate it into two feature columns and define y=3x. The columns are perfectly correlated and contain the same information. A no-intercept prediction depends only on the sum of their coefficients.

Inspect what each penalty prefers

For ridge with alpha1 under the squared-error-sum convention, the fitted coefficients are both10/7, approximately1.428571. Its predicted slope is20/7, approximately2.857143.

For lasso with alpha0.5 under its averaged half-squared-error convention, the demonstrated cyclic solver returns coefficients(2.75,0). It selects one duplicate column even though the other contains exactly the same information.

The Ridge and Lasso references specify their objectives. These numerical alpha values are not directly comparable because the loss normalization and penalty form differ.

Verify the reference solutions

import numpy as np
from linear_cases import duplicate_predictors

r = duplicate_predictors()
assert np.allclose(r['ridge']['coefficients'],[10/7,10/7])
assert np.allclose(r['lasso']['coefficients'],[2.75,0],atol=1e-10)
assert np.isclose(r['ridge']['prediction_at_x1'],20/7)
assert np.isclose(r['lasso']['prediction_at_x1'],2.75)
x=np.arange(-2.,3.)
assert np.allclose(2.75*x,1.375*x+1.375*x)
assert abs(2.75)+abs(0)==abs(1.375)+abs(1.375)
assert 1.375**2+1.375**2 < 2.75**2
print(r)

Run from the supervised-model lab. Feature units are unchanged and the intercept is disabled, keeping the small calculation transparent.

Explain the apparent feature selection

For a fixed positive coefficient sum, distributing it across the duplicate columns leaves predictions and the L1 norm unchanged. The lasso objective therefore does not uniquely identify how that positive sum should be allocated. The cyclic solver's returned sparse solution is one valid allocation.

An L2 penalty is smaller when a fixed sum is shared equally, so ridge has a preference for the balanced solution. This explains coefficient behavior in the exact duplicate case without attributing causal meaning to either feature.

With near duplicates, numerical differences and sample variation can affect which features lasso selects. Investigate selection stability across appropriate development samples rather than treating one fitted support set as definitive evidence.

Choose using the prediction and interpretation requirements

Ridge can retain correlated predictors while shrinking coefficients. Lasso can produce sparse representations, but sparsity is not automatically interpretability when features substitute for one another.

Standardization changes the penalty's relationship to raw feature units. Fit any learned scaling inside training folds and compare candidate penalties under an appropriate validation design. Document the objective normalization when moving between libraries or hand-written formulas.

This noiseless five-row example is a coefficient-behavior demonstration with no held-out performance comparison. Its training errors do not establish that ridge or lasso is the better predictive method for another task.

Exercise: add a small perturbation to the second duplicate column and repeat the fit across several prespecified perturbation seeds. Record coefficients and predictions separately. Explain why unstable individual coefficients can coexist with relatively stable predictions on the observed feature combinations.

NeuraPath's Data Science course connects regularization with feature dependence. A defensible explanation distinguishes a penalty's preference from evidence that one correlated feature is uniquely responsible for an outcome.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Logistic regression: probabilities, logits and decision thresholds.
• Continue with Elastic net: combine shrinkage and feature selection.