PCA

Introduction

Principal Component Analysis (PCA) is an unsupervised dimensionality reduction technique. Its goal is to find a low-dimensional representation of the data that captures the maximum amount of variance.

Key applications include:

Data visualization (projecting high-dimensional data to 2D or 3D)
Noise reduction
Preprocessing and feature extraction for downstream models
Exploratory data analysis

The Idea

PCA finds directions (called principal components) along which the data varies the most.

The first PC is the direction of maximum variance in the data.
The second PC is the direction of maximum variance orthogonal to the first.
And so on for subsequent components.

Each principal component is a linear combination of the original variables:

\[ Z_k = w_{k1} X_1 + w_{k2} X_2 + \cdots + w_{kp} X_p = \mathbf{w}_k^\top \mathbf{X} \]

The weights \(\mathbf{w}_k\) (also called loadings) define the direction of the \(k\)-th principal component.

Mathematical Formulation

Given a data matrix \(\mathbf{X}\) of size \(n \times p\), first center the data by subtracting column means:

\[ \mathbf{X}_c = \mathbf{X} - \mathbf{1}\bar{\mathbf{x}}^\top \]

The sample covariance matrix is:

\[ \mathbf{S} = \frac{1}{n-1} \mathbf{X}_c^\top \mathbf{X}_c \]

The principal components are the eigenvectors of \(\mathbf{S}\), and the eigenvalues represent the variance explained by each component.

First Principal Component

The first PC solves the optimization problem:

\[ \mathbf{w}_1 = \arg\max_{\|\mathbf{w}\|=1} \mathbf{w}^\top \mathbf{S} \mathbf{w} \]

By the method of Lagrange multipliers, the solution satisfies the eigenvalue equation:

\[ \mathbf{S} \mathbf{w}_1 = \lambda_1 \mathbf{w}_1 \]

where \(\mathbf{w}_1\) is the eigenvector corresponding to the largest eigenvalue \(\lambda_1\).

k-th Principal Component

The \(k\)-th PC is the eigenvector corresponding to the \(k\)-th largest eigenvalue:

\[ \mathbf{S} \mathbf{w}_k = \lambda_k \mathbf{w}_k, \quad \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0 \]

The eigenvectors are orthogonal: \(\mathbf{w}_i^\top \mathbf{w}_j = 0\) for \(i \neq j\).

Singular Value Decomposition (SVD)

In practice, PCA is computed via the SVD of the centered data matrix:

\[ \mathbf{X}_c = \mathbf{U} \mathbf{D} \mathbf{V}^\top \]

where:

\(\mathbf{U}\) is \(n \times p\) with orthonormal columns (left singular vectors)
\(\mathbf{D}\) is \(p \times p\) diagonal with singular values \(d_1 \geq d_2 \geq \cdots \geq d_p \geq 0\)
\(\mathbf{V}\) is \(p \times p\) orthogonal (right singular vectors)

The columns of \(\mathbf{V}\) are the principal components (loadings). The relationship between singular values and eigenvalues is:

\[ \lambda_k = \frac{d_k^2}{n-1} \]

The scores (projections of the data onto the PCs) are:

\[ \mathbf{Z} = \mathbf{X}_c \mathbf{V} = \mathbf{U} \mathbf{D} \]

SVD is preferred over eigendecomposition of the covariance matrix because it is more numerically stable, especially when \(p\) is large relative to \(n\).

Proportion of Variance Explained

The proportion of variance explained (PVE) by the \(k\)-th component is:

\[ \text{PVE}_k = \frac{\lambda_k}{\sum_{j=1}^{p} \lambda_j} \]

The cumulative proportion of variance explained by the first \(K\) components is:

\[ \text{CPVE}_K = \frac{\sum_{k=1}^{K} \lambda_k}{\sum_{j=1}^{p} \lambda_j} \]

Choosing the Number of Components

Common criteria:

Scree plot: plot eigenvalues \(\lambda_k\) vs. component number \(k\). Look for an "elbow" where the curve flattens.
Cumulative PVE threshold: retain enough components so that \(\text{CPVE}_K \geq 80\%\text{–}90\%\).
Kaiser's rule: retain components with \(\lambda_k > 1\) (when using the correlation matrix).

Loadings and Interpretation

The loading of variable \(j\) on principal component \(k\) is the element \(v_{jk}\) of the eigenvector \(\mathbf{w}_k\):

\[ \text{Loading}_{jk} = v_{jk} \]

A high absolute loading means that variable \(j\) contributes strongly to PC \(k\).
The sign indicates direction of contribution.

Biplot

A biplot displays both observations and variables in the space of the first two PCs:

Points represent observations projected onto PC1 and PC2 (scores).
Arrows represent loading vectors for each original variable.
The angle between arrows approximates the correlation between variables.
The length of an arrow indicates how well the variable is represented in the 2D projection.

Standardization

If the original variables are measured on different scales, PCA on the covariance matrix will be dominated by variables with the largest variance. In this case, it is recommended to use the correlation matrix instead:

\[ \mathbf{R} = \mathbf{D}_s^{-1} \mathbf{S} \mathbf{D}_s^{-1} \]

where \(\mathbf{D}_s = \text{diag}(s_1, \ldots, s_p)\) is the diagonal matrix of standard deviations. This is equivalent to standardizing each variable to zero mean and unit variance before applying PCA.

Using the correlation matrix is generally recommended unless the variables are already on the same scale and the relative magnitudes of variance are meaningful.

Limitations

Linear only: PCA can only capture linear relationships. For nonlinear structure, consider kernel PCA or autoencoders.
Sensitive to outliers: Outliers can dominate the covariance matrix and distort the principal components. Robust PCA methods exist for this case.
Interpretability: Principal components are linear combinations of all original variables, so they may not correspond to interpretable features.
Variance ≠ information: PCA assumes that directions of maximum variance are the most informative. This is not always true (e.g., in classification problems where discriminative directions may have low variance).