Ridge regression, also known as Tikhonov regularization or L2 regularization, was proposed by Hoerl and Kennard (1970). It addresses two fundamental problems in linear regression: multicollinearity and overfitting, by adding an L2 penalty term to the ordinary least squares objective.
Ridge regression is particularly useful when the number of predictors \(p\) is close to or larger than the number of observations \(n\), a setting where OLS either becomes unstable or is undefined altogether.
The ridge estimator minimizes the penalized residual sum of squares:
\[ \hat{\beta}_{ridge} = \arg\min_{\beta} \left\{ \|y - X\beta\|^2 + \lambda \|\beta\|_2^2 \right\} \]
Expanding:
\[ \sum_{i=1}^{n} (y_i - x_i'\beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \]
The penalty term \(\lambda \|\beta\|_2^2\) penalizes large coefficients, shrinking them toward zero. The regularization parameter \(\lambda \geq 0\) controls the trade-off:
Taking the derivative and setting it to zero yields the closed-form solution:
\[ \hat{\beta}_{ridge} = (X'X + \lambda I)^{-1} X'y \]
This solution always exists for \(\lambda > 0\). The matrix \(X'X + \lambda I\) is always invertible because adding \(\lambda > 0\) to the diagonal ensures all eigenvalues are strictly positive, even when \(X'X\) is singular.
Contrast with OLS:
\[ \hat{\beta}_{OLS} = (X'X)^{-1} X'y \]
which fails when \(X'X\) is singular (i.e., when columns of \(X\) are linearly dependent or \(p > n\)).
The mean squared error of the ridge estimator decomposes as:
\[ E\left[\|\hat{\beta}_{ridge} - \beta_{true}\|^2\right] = \text{Bias}^2 + \text{Variance} \]
\[ \text{Bias}(\hat{\beta}_{ridge}) = E[\hat{\beta}_{ridge}] - \beta_{true} = -\lambda(X'X + \lambda I)^{-1}\beta_{true} \]
The bias increases with \(\lambda\): larger penalty means more shrinkage away from the true value.
\[ \text{Var}(\hat{\beta}_{ridge}) = \sigma^2 \, \text{tr}\left[(X'X + \lambda I)^{-1} X'X (X'X + \lambda I)^{-1}\right] \]
The variance decreases with \(\lambda\): the penalty stabilizes the estimates.
There exists an optimal \(\lambda^*\) that minimizes total MSE — the point where the decrease in variance outweighs the increase in bias.
Let the Singular Value Decomposition of \(X\) be:
\[ X = UDV' \]
where \(D = \text{diag}(d_1, \ldots, d_p)\) contains the singular values. Then the ridge estimator can be written as:
\[ \hat{\beta}_{ridge} = V \, \text{diag}\!\left(\frac{d_j^2}{d_j^2 + \lambda}\right) V' \hat{\beta}_{OLS} \]
Each OLS coefficient (in the rotated coordinate system) is shrunk by a factor:
\[ \frac{d_j^2}{d_j^2 + \lambda} \]
Directions with small singular values (high variance directions) are shrunk the most. Directions with large singular values are barely affected.
\[ \text{df}(\lambda) = \sum_{j=1}^{p} \frac{d_j^2}{d_j^2 + \lambda} \]
When \(\lambda = 0\), \(\text{df} = p\) (full OLS). As \(\lambda \to \infty\), \(\text{df} \to 0\).
\[ \text{GCV}(\lambda) = \frac{\frac{1}{n}\|y - X\hat{\beta}_{ridge}\|^2}{\left(1 - \frac{\text{df}(\lambda)}{n}\right)^2} \]
GCV is computationally efficient as it avoids refitting for each fold.
Split data into \(K\) folds, fit ridge on \(K-1\) folds, evaluate prediction error on the held-out fold. Choose \(\lambda\) that minimizes average CV error.
Plot the coefficient values \(\hat{\beta}_j(\lambda)\) as a function of \(\lambda\). Look for a region where coefficients stabilize — this indicates an appropriate level of regularization.
When predictors are highly correlated, the matrix \(X'X\) has near-zero eigenvalues. This makes OLS coefficients unstable: small changes in the data lead to large changes in estimates (huge variance).
Ridge regression stabilizes by adding \(\lambda\) to every eigenvalue:
\[ \text{eigenvalues of } (X'X + \lambda I) = d_j^2 + \lambda \]
Even if \(d_j^2 \approx 0\), the term \(d_j^2 + \lambda\) remains bounded away from zero, preventing the explosion of the inverse.
A common diagnostic for multicollinearity:
\[ \text{VIF}_j = \frac{1}{1 - R_j^2} \]
where \(R_j^2\) is the \(R^2\) from regressing predictor \(j\) on all other predictors. A VIF > 10 suggests problematic multicollinearity.
Ridge regression is equivalent to the Maximum A Posteriori (MAP) estimate under a Gaussian prior on the coefficients:
\[ \beta \sim \mathcal{N}\!\left(0, \, \frac{\sigma^2}{\lambda} I\right) \]
The prior precision (inverse variance) is \(\frac{\lambda}{\sigma^2}\). A larger \(\lambda\) corresponds to a tighter prior around zero — encoding the belief that coefficients should be small.
The posterior mode of:
\[ p(\beta | y, X) \propto \exp\!\left( -\frac{1}{2\sigma^2} \|y - X\beta\|^2 \right) \cdot \exp\!\left( -\frac{\lambda}{2\sigma^2} \|\beta\|^2 \right) \]
is exactly \(\hat{\beta}_{ridge}\).