Discriminant Analysis is a classification method based on the generative approach: instead of directly modeling the posterior \(P(Y=k|X)\) (as discriminative methods like logistic regression do), it models the class-conditional density \(P(X|Y=k)\) for each class \(k\), and then applies Bayes' theorem to obtain the posterior:
\[ P(Y=k \mid X=x) = \frac{P(X=x \mid Y=k) \, P(Y=k)}{P(X=x)} = \frac{\pi_k \, f_k(x)}{\sum_{l=1}^{K} \pi_l \, f_l(x)} \]
where \(\pi_k = P(Y=k)\) is the prior probability of class \(k\), and \(f_k(x) = P(X=x \mid Y=k)\) is the class-conditional density.
The key assumption is that each class-conditional distribution is multivariate normal:
\[ f_k(x) = \frac{1}{(2\pi)^{p/2} |\Sigma_k|^{1/2}} \exp\left( -\frac{1}{2}(x - \mu_k)^\top \Sigma_k^{-1} (x - \mu_k) \right) \]
The classification rule assigns observation \(x\) to the class \(k\) that maximizes \(P(Y=k|X=x)\), or equivalently maximizes \(\pi_k \, f_k(x)\).
LDA assumes that all classes share the same covariance matrix \(\Sigma\):
\[ X \mid Y=k \sim \mathcal{N}(\mu_k, \Sigma) \quad \text{for all } k = 1, \ldots, K \]
By Bayes' theorem, the posterior is proportional to:
\[ P(Y=k \mid X=x) \propto \pi_k \, f_k(x) \]
Taking the log and dropping terms that do not depend on \(k\) (since \(\Sigma\) is shared), we obtain the linear discriminant function:
\[ \delta_k(x) = x^\top \Sigma^{-1} \mu_k - \frac{1}{2} \mu_k^\top \Sigma^{-1} \mu_k + \log(\pi_k) \]
The classification rule is: assign \(x\) to the class \(k\) with the largest \(\delta_k(x)\).
Since \(\delta_k(x)\) is a linear function of \(x\), the decision boundaries between any two classes are hyperplanes in the feature space.
When \(K=2\), the decision boundary is the set of points where \(\delta_1(x) = \delta_2(x)\):
\[ \{x : \delta_1(x) = \delta_2(x)\} \]
This defines a hyperplane in \(\mathbb{R}^p\). Classification reduces to checking the sign of \(\delta_1(x) - \delta_2(x)\).
Fisher's Linear Discriminant seeks the projection direction \(w\) that maximizes the ratio of between-class variance to within-class variance:
\[ \max_w \frac{w^\top S_B \, w}{w^\top S_W \, w} \]
where \(S_B = (\mu_1 - \mu_2)(\mu_1 - \mu_2)^\top\) is the between-class scatter matrix and \(S_W = \Sigma\) is the within-class scatter (pooled covariance).
The solution is:
\[ w = \Sigma^{-1}(\mu_1 - \mu_2) \]
This direction is the same as the one used in LDA's classification rule, confirming that LDA projects data onto the direction of maximum class separability.
QDA relaxes the equal covariance assumption, allowing each class to have its own covariance matrix:
\[ X \mid Y=k \sim \mathcal{N}(\mu_k, \Sigma_k) \]
Since \(\Sigma_k\) differs across classes, the log-likelihood terms involving the covariance no longer cancel. The quadratic discriminant function is:
\[ \delta_k(x) = -\frac{1}{2} \log|\Sigma_k| - \frac{1}{2}(x - \mu_k)^\top \Sigma_k^{-1}(x - \mu_k) + \log(\pi_k) \]
The classification rule remains the same: assign \(x\) to the class with the largest \(\delta_k(x)\).
Since \(\delta_k(x)\) is now a quadratic function of \(x\), decision boundaries are quadratic surfaces (ellipses, hyperbolas, or parabolas depending on the relative geometry of the covariance matrices).
The fundamental trade-off is bias vs variance:
Consequences:
Parameters are estimated from the training data using plug-in (maximum likelihood) estimates:
These estimates are plugged into the discriminant functions \(\delta_k(x)\) to obtain classification rules.
RDA provides a smooth interpolation between LDA and QDA by defining a regularized covariance estimate:
\[ \hat{\Sigma}_k(\alpha) = \alpha \, \hat{\Sigma}_k + (1 - \alpha) \, \hat{\Sigma} \]
where \(\alpha \in [0, 1]\) is a tuning parameter:
The optimal \(\alpha\) is chosen by cross-validation, balancing the bias of the equal-covariance assumption against the variance of estimating separate covariance matrices.
This approach is especially useful in high-dimensional settings where QDA's covariance estimates may be singular or poorly conditioned.
For the two-class problem, LDA is equivalent to linear regression of the class indicator variable on \(X\). Both methods produce the same classification boundary (though predicted values differ in interpretation).
Both LDA and logistic regression model the log-odds as a linear function of \(x\):
\[ \log \frac{P(Y=1|X=x)}{P(Y=2|X=x)} = \beta_0 + \beta^\top x \]
The difference lies in the assumptions:
When the normality assumption holds, LDA is more efficient (lower asymptotic variance). When it does not hold, logistic regression is more robust.
Naive Bayes assumes that features are conditionally independent given the class, i.e., \(\Sigma_k\) is diagonal. With Gaussian class-conditionals, Naive Bayes is equivalent to LDA with a diagonal covariance matrix:
\[ \Sigma = \text{diag}(\sigma_1^2, \sigma_2^2, \ldots, \sigma_p^2) \]
This drastically reduces the number of parameters from \(\frac{p(p+1)}{2}\) to \(p\), making it suitable for high-dimensional problems where full covariance estimation is infeasible.