Regularization as a constraint on model complexity

Regularization expresses a preference among fitted models, often by penalizing large coefficients. It changes the optimization problem. Whether that preference improves predictions must be evaluated on appropriate unseen data.

A tiny original regression example makes the tradeoff visible. Let x be 1, 2 and 3, and let y=2x. Fit a model prediction=theta*x with no intercept. The unpenalized coefficient is two and fits every training observation exactly.

Write the actual objective

The sum of squared residuals is 14*(theta-2)**2. Add the ridge penalty alpha*theta**2. With alpha=14, differentiating the full objective gives 28*(theta-2)+28*theta=0, so theta=1.

At this solution, the residual sum of squares is 14, the penalty is 14 and the total objective is 28. At theta=2, the residual is zero but the penalty is 56. The optimizer prefers theta=1 because it minimizes the declared penalized objective, even though training residual error is larger.

The scikit-learn Ridge documentation specifies a squared-error sum plus an L2 penalty. Check this convention before transferring an alpha value from another implementation.

Verify the closed form against the library

import numpy as np
from sklearn.linear_model import Ridge
from math_core import regression_data
from inference_core import ridge_reference

x,y = regression_data()
r = ridge_reference(alpha=14)
fit = Ridge(alpha=14,fit_intercept=False,solver='svd').fit(x[:,None],y)
assert np.isclose(fit.coef_[0],1)
assert r['theta']==1 and r['objective']==28
assert r['residual_term']==r['penalty']==14
assert ridge_reference(alpha=0)['theta']==2
assert ridge_reference(alpha=14,mean_loss=True)['theta']==.5
print({'SSE_convention':r,'MSE_same_alpha':ridge_reference(alpha=14,mean_loss=True)})

The included lab supplies the CSV, helper and recorded outputs. It uses scikit-learn 1.9.0 locally; the linked stable documentation may describe a newer patch release.

Why averaging the loss changes alpha's meaning

Dividing the residual sum by three while leaving alpha=14 unchanged strengthens the penalty relative to the data-fit term. The solution becomes theta=0.5. To preserve the original optimum, divide both the residual term and alpha by three.

This explains why identical regularization numbers need not represent identical constraints across libraries, dataset sizes or hand-written losses. Record whether the loss is summed or averaged, whether the intercept is penalized, and which coefficients receive the penalty.

Connect penalty and constraint carefully

A related formulation minimizes residual error subject to a bound on the squared coefficient norm. In convex ridge settings, penalty strength and an appropriate norm bound express corresponding tradeoffs. The bound is not numerically equal to alpha; it depends on the data and solution.

Feature units matter too. Changing a feature from one unit to another changes the coefficient required to represent the same predictions. An unadjusted coefficient penalty can therefore change the fitted model. Fit any learned scaling on training data and apply that fitted transformation to validation data.

Decide with validation, not smaller coefficients alone

This noiseless fixture has no generalization experiment. Shrinking the true coefficient from two to one worsens predictions under its exact y=2x rule. Regularization can be valuable with noisy or unstable estimation, but it does not guarantee improvement in every setting.

Exercise: compare alpha=0, 7 and 14 using the closed form. Report residual error and penalty separately. Then explain what additional split and evaluation evidence would be needed to choose alpha for a real predictive task.

NeuraPath's Data Science course connects model complexity with validation. A defensible regularization choice includes the objective convention, preprocessing scope and measured predictive tradeoff.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Maximum likelihood estimation with a coin example.
• Continue with Bias and variance with repeated training samples.