Elastic net: combine shrinkage and feature selection

Elastic net combines L1 and L2 coefficient penalties. The L1 component can set coefficients to zero, while the L2 component encourages shrinkage and can stabilize allocation among correlated predictors. The actual result depends on the data, feature scaling and chosen penalty parameters.

Our original example uses two identical signal columns x=(-2,-1,0,1,2), target y=3x, and a third feature(1,-2,2,-2,1). The third feature is orthogonal to x in this small fixture. No intercept or automatic scaling is used.

Write the implemented objective

Scikit-learn's ElasticNet reference specifies averaged half-squared-error loss plus alpha*l1_ratio*L1 and 0.5*alpha*(1-l1_ratio)*L2 penalties.

Here alpha=0.5 and l1_ratio=0.5. The L1 multiplier is0.25 and the squared-L2 multiplier0.125. Alpha controls overall strength, while l1_ratio controls the mixture. A mixing value of0.5 does not mean that the two realized penalty contributions must have equal numerical values.

Derive the symmetric signal solution

For equal positive coefficients b on the two duplicate signal columns, the fitted slope is2b. The loss and penalty reduce to (3-2b)**2 + 0.5*b + 0.25*b**2. Differentiation gives 8.5*b-11.5=0, so b=23/17, approximately1.352941.

The orthogonal third feature receives coefficient zero in the recorded fit. The two signal columns remain nonzero and share the coefficient. Thus this case demonstrates shrinkage and a sparse component without claiming that elastic net always selects exactly one member of a correlated group.

import numpy as np
from linear_cases import elastic_sparse_case

r = elastic_sparse_case()
assert np.allclose(r['coefficients'],[23/17,23/17,0],atol=1e-9)
assert r['unrelated_dot_target']==0
b=23/17
assert np.isclose(8.5*b-11.5,0)
assert np.isclose(r['training_mse'],2*(3-2*b)**2)
assert sum(abs(value)>1e-8 for value in r['coefficients'])==2
print(r)

Run in the supervised-model lab. The analytical check is separate from the numerical solver result. A tolerance is used for floating-point coefficients rather than demanding identical decimal strings.

Tune both parameters within the evaluation design

Alpha and l1_ratio should be selected using development evidence appropriate to the deployment population. Fit preprocessing within each training fold. Record the candidate grid and search metric, then evaluate the selected procedure on suitable held-out evidence.

Do not compare raw alpha values across ridge, lasso and elastic net without checking their loss conventions. The included ridge reference uses a summed residual loss, while this elastic-net implementation uses an averaged half-squared loss.

Interpret a zero coefficient cautiously

A zero coefficient means the fitted penalized model did not retain that feature under this representation and parameter choice. It does not prove that the feature is irrelevant under every model, population or interaction structure.

Our third feature was deliberately constructed to be orthogonal to the target in a noiseless linear case. Real data may contain nonlinear relationships or correlated substitutes that require a more careful feature-importance and stability analysis.

The five-row example has no held-out experiment and makes no prediction-quality claim. Its purpose is to explain the objective and coefficient behavior before using elastic net in a larger evaluation workflow.

Exercise: vary l1_ratio across0.2,0.5 and0.8 while holding alpha fixed. Report coefficient values, nonzero counts and the separate loss/penalty terms. Explain why choosing the sparsest fit alone is not a sufficient model-selection rule.

NeuraPath's Data Science course connects regularization mathematics with model evaluation. A clear elastic-net result reports the objective, feature representation and evidence supporting the selected tradeoff.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Ridge versus lasso when predictors are correlated.
• Continue with Decision tree depth: visualize overfitting on a small dataset.