Shrinkage Methods

Introduction

In high-dimensional settings where the number of predictors \(p\) is large relative to the number of observations \(n\), Ordinary Least Squares (OLS) tends to overfit the training data. Shrinkage methods constrain or regularize the coefficient estimates, shrinking them toward zero. This reduces variance at the cost of a slight increase in bias.

The two main shrinkage approaches are:

Ridge Regression (L2 penalty)
LASSO (L1 penalty)

Both add a penalty term to the OLS objective function, controlling model complexity through a tuning parameter \(\lambda\).

Why Shrink?

The motivation comes from the bias-variance tradeoff. Recall that the expected prediction error (MSE) decomposes as:

\[ \text{MSE} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error} \]

OLS is unbiased, but when \(p\) is large (especially when \(p \approx n\) or \(p > n\)), the variance of the coefficient estimates becomes very high. By accepting a small amount of bias, we can substantially reduce variance, leading to lower overall MSE.

Geometric intuition: Shrinkage methods solve a constrained optimization problem. Instead of finding the OLS solution anywhere in parameter space, we restrict the coefficients to lie within a constraint region (a sphere for Ridge, a diamond for LASSO). The solution is the point where the RSS contours first touch the constraint region.

Ridge Regression (L2 Penalty)

Ridge regression minimizes the penalized residual sum of squares:

\[ \hat{\beta}^{\text{ridge}} = \underset{\beta}{\arg\min} \left\{ \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij} \beta_j \right)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \right\} \]

This is equivalent to the constrained problem:

\[ \min_{\beta} \; \text{RSS} \quad \text{subject to} \quad \sum_{j=1}^{p} \beta_j^2 \leq s \]

There is a one-to-one correspondence between \(\lambda\) and \(s\).

Closed-Form Solution

Ridge regression has an explicit solution in matrix form:

\[ \hat{\beta}^{\text{ridge}} = (X^T X + \lambda I)^{-1} X^T y \]

Note that \(X^T X + \lambda I\) is always invertible for \(\lambda > 0\), even when \(X^T X\) is singular (which happens when \(p > n\)).

Properties

Shrinks all coefficients toward zero, but never sets them exactly to zero
When \(\lambda = 0\): reduces to OLS
When \(\lambda \to \infty\): all \(\hat{\beta}_j \to 0\)
The intercept \(\beta_0\) is typically not penalized
Important: Predictors must be standardized before applying Ridge, since the penalty is not scale-invariant

LASSO (L1 Penalty)

The LASSO (Least Absolute Shrinkage and Selection Operator) minimizes:

\[ \hat{\beta}^{\text{lasso}} = \underset{\beta}{\arg\min} \left\{ \sum_{i=1}^{n} \left( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij} \beta_j \right)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \right\} \]

Equivalently:

\[ \min_{\beta} \; \text{RSS} \quad \text{subject to} \quad \sum_{j=1}^{p} |\beta_j| \leq s \]

Key Difference from Ridge

The L1 penalty can set coefficients exactly to zero. This means LASSO performs variable selection — it produces sparse models where only a subset of predictors have non-zero coefficients.

There is no closed-form solution for the LASSO. It is typically solved using coordinate descent algorithms.

Geometric Interpretation

The L1 constraint region is a diamond (in 2D) or a cross-polytope (in higher dimensions). Its corners lie on the coordinate axes. The RSS contours (ellipses) are most likely to first touch the constraint region at a corner, where one or more coordinates are exactly zero. In contrast, the L2 constraint region is a sphere with no corners, so tangency points rarely occur on the axes.

Elastic Net

The Elastic Net combines both L1 and L2 penalties:

\[ \hat{\beta}^{\text{enet}} = \underset{\beta}{\arg\min} \left\{ \text{RSS} + \lambda \left[ \alpha \sum_{j=1}^{p} |\beta_j| + (1-\alpha) \sum_{j=1}^{p} \beta_j^2 \right] \right\} \]

where \(\alpha \in [0,1]\) controls the mix:

\(\alpha = 1\): pure LASSO
\(\alpha = 0\): pure Ridge

When to use Elastic Net:

When predictors are highly correlated. In this case, LASSO tends to arbitrarily select one variable from a group of correlated predictors and discard the others. Elastic Net encourages a grouping effect — correlated predictors tend to get similar coefficients.
When \(p > n\): LASSO can select at most \(n\) variables, while Elastic Net does not have this limitation.

Choosing \(\lambda\)

The tuning parameter \(\lambda\) controls the strength of regularization. It is chosen via cross-validation:

Define a grid of \(\lambda\) values (typically on a log scale)
For each \(\lambda\), perform K-fold CV and compute the average prediction error
Select the \(\lambda\) that minimizes cross-validated error

Selection Rules

\(\lambda_{\min}\): the value that minimizes the CV error — gives the best predictive model
\(\lambda_{1\text{se}}\): the largest \(\lambda\) within one standard error of the minimum — gives the most parsimonious model whose error is statistically indistinguishable from the best

Regularization Path

A useful diagnostic is the regularization path: a plot of coefficient values as a function of \(\lambda\) (or equivalently, of \(\|\hat{\beta}\|_1 / \|\hat{\beta}^{\text{OLS}}\|_1\)). This shows how each coefficient is progressively shrunk toward zero as regularization increases.

Comparison: Ridge vs LASSO

	Ridge (L2)	LASSO (L1)
Penalty	\(\lambda \sum \beta_j^2\)	\(\lambda \sum \|\beta_j\|\)
Variable selection	No (all coefficients remain non-zero)	Yes (sets some to exactly zero)
Closed-form	Yes	No
Best when	Many predictors with small effects	Few predictors with large effects (sparse truth)
Correlated predictors	Shares weight among group	Selects one arbitrarily
Constraint geometry	Sphere (smooth)	Diamond (corners on axes)

Summary: Both Ridge and LASSO outperform OLS when \(p\) is large relative to \(n\). Ridge is preferable when the true model has many small, diffuse effects. LASSO is preferable when only a few predictors are truly relevant (sparse models). When in doubt, Elastic Net provides a compromise.