Let \(X_1, X_2, \ldots, X_n\) be a random sample from a distribution with density \(p(x; \theta)\), where \(\theta \in \Theta\) is an unknown parameter. The likelihood function is defined as the joint density evaluated at the observed data, viewed as a function of \(\theta\):
\[ L(\theta; \mathbf{x}) = \prod_{i=1}^{n} p(x_i; \theta) \]
The likelihood is not a probability distribution over \(\theta\); it measures how well a particular value of \(\theta\) explains the observed data. Two likelihood functions \(L_1\) and \(L_2\) carry the same information about \(\theta\) if \(L_1(\theta; \mathbf{x}) = c \cdot L_2(\theta; \mathbf{x})\) for some constant \(c > 0\) not depending on \(\theta\).
The log-likelihood function is the natural logarithm of the likelihood:
\[ \ell(\theta) = \log L(\theta; \mathbf{x}) = \sum_{i=1}^{n} \log p(x_i; \theta) \]
Since the logarithm is a strictly increasing function, maximizing \(\ell(\theta)\) is equivalent to maximizing \(L(\theta; \mathbf{x})\). The log-likelihood transforms products into sums, making analytical and numerical optimization tractable.
For comparing two nested models \(M_0 \subset M_1\), the likelihood ratio is:
\[ \Lambda(\mathbf{x}) = \frac{\sup_{\theta \in \Theta_0} L(\theta; \mathbf{x})}{\sup_{\theta \in \Theta} L(\theta; \mathbf{x})} = \frac{L(\hat{\theta}_0; \mathbf{x})}{L(\hat{\theta}; \mathbf{x})} \]
where \(\hat{\theta}_0\) is the MLE under the restricted model and \(\hat{\theta}\) is the unrestricted MLE. Note that \(0 \leq \Lambda(\mathbf{x}) \leq 1\), and small values of \(\Lambda\) provide evidence against \(M_0\). Under regularity conditions and \(H_0\) true:
\[ -2 \log \Lambda(\mathbf{x}) \xrightarrow{d} \chi^2_m \]
where \(m = \dim(\Theta) - \dim(\Theta_0)\) is the difference in the number of free parameters.
A statistic \(S = t(\mathbf{X})\) is sufficient for \(\theta\) if the conditional distribution of the data given \(S\) does not depend on \(\theta\):
\[ g(\mathbf{x} | S = s; \theta) = g(\mathbf{x} | S = s) \quad \forall \theta \in \Theta \]
Intuitively, a sufficient statistic captures all the information about \(\theta\) contained in the sample. Once we know the value of a sufficient statistic, the remaining variation in the data is pure noise — uninformative about the parameter.
A statistic \(S = t(\mathbf{X})\) is sufficient for \(\theta\) if and only if the likelihood can be factored as:
\[ L(\theta; \mathbf{x}) = g(t(\mathbf{x}); \theta) \cdot h(\mathbf{x}) \]
where \(g\) depends on the data only through \(t(\mathbf{x})\), and \(h\) does not depend on \(\theta\). This theorem provides a practical criterion for identifying sufficient statistics without computing conditional distributions directly.
A sufficient statistic \(S\) is minimal sufficient if it is a function of every other sufficient statistic. Equivalently, \(S\) achieves the maximum possible data reduction while retaining all information about \(\theta\). A sufficient statistic \(t(\mathbf{X})\) is minimal sufficient if and only if:
\[ \frac{L(\theta; \mathbf{x})}{L(\theta; \mathbf{y})} \text{ does not depend on } \theta \iff t(\mathbf{x}) = t(\mathbf{y}) \]
The minimal sufficient statistic is essentially unique (up to one-to-one transformations).
A parametric family of distributions belongs to the exponential family if its likelihood can be written in the form:
\[ L(\mathbf{x}; \theta) = h(\mathbf{x}) \cdot \exp\left[ \sum_{j=1}^{k} g_j(\theta) \, s_j(\mathbf{x}) - k(\theta) \right] \]
where:
By the Factorization Theorem, the vector \(\mathbf{S} = (s_1(\mathbf{x}), s_2(\mathbf{x}), \ldots, s_k(\mathbf{x}))\) is sufficient for \(\theta\). Moreover, if the natural parameter space contains an open set in \(\mathbb{R}^k\), then \(\mathbf{S}\) is the minimal sufficient statistic.
This is a key result: for exponential families, the dimensionality of the minimal sufficient statistic equals the number of free parameters, regardless of sample size.
Normal: \(s_1(\mathbf{x}) = \sum x_i\), \(s_2(\mathbf{x}) = \sum x_i^2\) are jointly sufficient for \((\mu, \sigma^2)\).
Poisson: \(s(\mathbf{x}) = \sum x_i\) is sufficient for \(\lambda\).
Bernoulli: \(s(\mathbf{x}) = \sum x_i\) is sufficient for \(p\).
An estimator \(T(\mathbf{X})\) is a function of the data used to estimate a parameter \(\tau(\theta)\). We evaluate estimators according to several criteria.
An estimator \(T(\mathbf{X})\) is unbiased for \(\tau(\theta)\) if:
\[ E_\theta[T(\mathbf{X})] = \tau(\theta) \quad \forall \theta \in \Theta \]
The bias of an estimator is \(b(\theta) = E_\theta[T(\mathbf{X})] - \tau(\theta)\). An unbiased estimator has \(b(\theta) = 0\) for all \(\theta\).
The mean squared error (MSE) decomposes as:
\[ \text{MSE}_\theta(T) = E_\theta\left[(T(\mathbf{X}) - \tau(\theta))^2\right] = \text{Var}_\theta(T) + [b(\theta)]^2 \]
For unbiased estimators, MSE equals variance. An estimator \(T_1\) is more efficient than \(T_2\) if \(\text{MSE}_\theta(T_1) \leq \text{MSE}_\theta(T_2)\) for all \(\theta\), with strict inequality for at least one value.
An estimator \(T_n = T_n(\mathbf{X})\) is consistent for \(\tau(\theta)\) if:
\[ T_n \xrightarrow{P} \tau(\theta) \quad \text{as } n \to \infty \]
A sufficient condition for consistency is that both \(\text{Var}_\theta(T_n) \to 0\) and \(b_n(\theta) \to 0\) as \(n \to \infty\).
The score function is the derivative of the log-likelihood with respect to \(\theta\):
\[ s(\theta; \mathbf{x}) = \frac{\partial}{\partial \theta} \ell(\theta; \mathbf{x}) = \ell'(\theta) \]
Under regularity conditions, \(E_\theta[s(\theta; \mathbf{X})] = 0\). The Fisher information is defined as the variance of the score:
\[ I_{\mathbf{X}}(\theta) = E_\theta\left[\left(\ell'(\theta)\right)^2\right] = -E_\theta\left[\ell''(\theta)\right] \]
The equality between the two forms holds under regularity conditions that allow interchange of differentiation and integration.
Fisher information measures the curvature of the log-likelihood function at its maximum. High curvature (high information) means the likelihood is sharply peaked around the MLE, so the data are highly informative about \(\theta\). Low curvature (low information) means a flat likelihood and greater uncertainty.
For a sample of \(n\) i.i.d. observations, information is additive:
\[ I_{\mathbf{X}}(\theta) = n \cdot I_{X_1}(\theta) \]
When \(\theta = (\theta_1, \ldots, \theta_p)^T\) is a vector, the Fisher information becomes a \(p \times p\) matrix:
\[ [I(\theta)]_{jk} = -E_\theta\left[\frac{\partial^2 \ell(\theta)}{\partial \theta_j \partial \theta_k}\right] = E_\theta\left[\frac{\partial \ell}{\partial \theta_j} \cdot \frac{\partial \ell}{\partial \theta_k}\right] \]
The Cramér–Rao inequality establishes a lower bound on the variance of any unbiased estimator.
If \(T(\mathbf{X})\) is an unbiased estimator of \(\theta\), then under regularity conditions:
\[ \text{Var}_\theta(T(\mathbf{X})) \geq \frac{1}{I_{\mathbf{X}}(\theta)} \]
An unbiased estimator that attains this lower bound is called efficient (or UMVUE for the class of unbiased estimators). It achieves the minimum possible variance.
For estimating a function \(\tau(\theta)\) with an estimator \(T(\mathbf{X})\) that may be biased, the general Cramér–Rao bound is:
\[ \text{Var}_\theta(T(\mathbf{X})) \geq \frac{[\tau'(\theta)]^2}{I_{\mathbf{X}}(\theta)} \]
where \(\tau'(\theta) = \frac{d}{d\theta}\tau(\theta)\). For the unbiased case (\(\tau(\theta) = \theta\)), we recover \(\tau'(\theta) = 1\) and the simpler form above.
The Cramér–Rao bound is attained if and only if the score function is a linear function of the estimator:
\[ \ell'(\theta; \mathbf{x}) = a(\theta) \left[T(\mathbf{x}) - \tau(\theta)\right] \]
This occurs naturally in exponential family distributions. If the bound is attained, then \(T\) is necessarily the UMVUE for \(\tau(\theta)\).
The Rao–Blackwell theorem provides a method for improving estimators using sufficient statistics.
Theorem. Let \(T(\mathbf{X})\) be any unbiased estimator of \(\tau(\theta)\), and let \(S = t(\mathbf{X})\) be a sufficient statistic for \(\theta\). Define:
\[ T^*(S) = E[T(\mathbf{X}) \mid S] \]
Then:
The variance reduction follows from the law of total variance:
\[ \text{Var}(T) = E[\text{Var}(T|S)] + \text{Var}(E[T|S]) = E[\text{Var}(T|S)] + \text{Var}(T^*) \]
Since \(E[\text{Var}(T|S)] \geq 0\), we have \(\text{Var}(T^*) \leq \text{Var}(T)\), with equality if and only if \(T\) is already a function of \(S\).
The process of conditioning on a sufficient statistic to improve an estimator is called Rao–Blackwellization.
The Lehmann–Scheffé theorem characterizes the unique best unbiased estimator.
A sufficient statistic \(S\) is complete if for any measurable function \(g\):
\[ E_\theta[g(S)] = 0 \quad \forall \theta \in \Theta \implies g(S) = 0 \text{ a.s.} \]
Completeness means that the only unbiased estimator of zero based on \(S\) is the trivial one. In exponential families with natural parameter space containing an open set, the natural sufficient statistic is complete.
Theorem (Lehmann–Scheffé). Let \(S\) be a complete sufficient statistic for \(\theta\). If \(T^* = h(S)\) is any unbiased estimator of \(\tau(\theta)\) that depends on the data only through \(S\), then \(T^*\) is the unique Uniformly Minimum Variance Unbiased Estimator (UMVUE) of \(\tau(\theta)\).
The proof relies on completeness: if two unbiased estimators \(T_1^*\) and \(T_2^*\) are both functions of \(S\), then \(E_\theta[T_1^* - T_2^*] = 0\) for all \(\theta\), and completeness implies \(T_1^* = T_2^*\) a.s.
Two strategies:
The maximum likelihood estimator (MLE) is defined as the value of \(\theta\) that maximizes the likelihood function:
\[ \hat{\theta}_{\text{MLE}} = \arg\max_{\theta \in \Theta} L(\theta; \mathbf{x}) = \arg\max_{\theta \in \Theta} \ell(\theta; \mathbf{x}) \]
Under regularity conditions, the MLE is found by solving the score equation:
\[ \ell'(\theta; \mathbf{x}) = \frac{\partial}{\partial \theta} \ell(\theta; \mathbf{x}) = 0 \]
1. Equivariance (Invariance): If \(\hat{\theta}\) is the MLE of \(\theta\), then for any function \(\psi = f(\theta)\), the MLE of \(\psi\) is \(\hat{\psi} = f(\hat{\theta})\).
2. Function of the sufficient statistic: In exponential families, the MLE is always a function of the minimal sufficient statistic.
3. Consistency: Under regularity conditions, \(\hat{\theta}_n \xrightarrow{P} \theta_0\) as \(n \to \infty\).
4. Asymptotic normality: Under regularity conditions:
\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N\left(0, \frac{1}{I_{X_1}(\theta_0)}\right) \]
Equivalently, for finite \(n\):
\[ \hat{\theta}_n \stackrel{\text{approx}}{\sim} N\left(\theta_0, \frac{1}{I_{\mathbf{X}}(\theta_0)}\right) = N\left(\theta_0, \frac{1}{n \cdot I_{X_1}(\theta_0)}\right) \]
5. Asymptotic efficiency: The MLE achieves the Cramér–Rao lower bound asymptotically. No consistent estimator has a smaller asymptotic variance.
The MLE is generally not unbiased in finite samples. For example, the MLE of \(\sigma^2\) in the normal model is \(\frac{1}{n}\sum(X_i - \bar{X})^2\), which has expectation \(\frac{n-1}{n}\sigma^2\). However, the bias vanishes asymptotically: \(b_n(\theta) = O(1/n)\).
The delta method provides the asymptotic distribution of a smooth function of an asymptotically normal estimator.
Theorem. If \(\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{d} N(0, \sigma^2)\) and \(\psi = f(\theta)\) is a differentiable function with \(f'(\theta) \neq 0\), then:
\[ \sqrt{n}(\hat{\psi}_n - \psi) = \sqrt{n}(f(\hat{\theta}_n) - f(\theta)) \xrightarrow{d} N\left(0, [f'(\theta)]^2 \sigma^2\right) \]
Applied to the MLE with \(\sigma^2 = 1/I_{X_1}(\theta)\):
\[ \sqrt{n}(\hat{\psi}_n - \psi) \xrightarrow{d} N\left(0, \frac{[f'(\theta)]^2}{I_{X_1}(\theta)}\right) \]
This result is fundamental for constructing confidence intervals and hypothesis tests for transformed parameters. The asymptotic variance of \(\hat{\psi}_n\) is:
\[ \text{Var}(\hat{\psi}_n) \approx \frac{[f'(\theta)]^2}{n \cdot I_{X_1}(\theta)} \]
If \(\sqrt{n}(\hat{\boldsymbol{\theta}}_n - \boldsymbol{\theta}) \xrightarrow{d} N(\mathbf{0}, \Sigma)\) and \(\psi = f(\boldsymbol{\theta})\) with gradient \(\nabla f(\boldsymbol{\theta})\), then:
\[ \sqrt{n}(f(\hat{\boldsymbol{\theta}}_n) - f(\boldsymbol{\theta})) \xrightarrow{d} N\left(0, \nabla f(\boldsymbol{\theta})^T \Sigma \, \nabla f(\boldsymbol{\theta})\right) \]
Consider testing \(H_0: \theta = \theta_0\) against \(H_1: \theta \neq \theta_0\). The three classical asymptotic tests are the Wald test, the likelihood ratio test, and the score (Rao) test. All three are asymptotically equivalent under \(H_0\).
The Wald test is based on the asymptotic normality of the MLE. The test statistic is:
\[ W(\mathbf{x}) = 2\left[\ell(\hat{\theta}) - \ell(\theta_0)\right] \]
Under \(H_0\) and regularity conditions:
\[ W(\mathbf{x}) \xrightarrow{d} \chi^2_m \]
where \(m\) is the number of constraints imposed by \(H_0\) (i.e., the dimension of the parameter space reduction). We reject \(H_0\) at level \(\alpha\) if \(W(\mathbf{x}) > \chi^2_{m,\alpha}\).
An equivalent formulation of the Wald test directly uses the standardized distance of the MLE from the null value:
\[ W^*(\mathbf{x}) = (\hat{\theta} - \theta_0)^T I_{\mathbf{X}}(\hat{\theta}) (\hat{\theta} - \theta_0) \xrightarrow{d} \chi^2_m \]
In the scalar case:
\[ W^*(\mathbf{x}) = \frac{(\hat{\theta} - \theta_0)^2}{\text{Var}(\hat{\theta})} = (\hat{\theta} - \theta_0)^2 \cdot I_{\mathbf{X}}(\hat{\theta}) \xrightarrow{d} \chi^2_1 \]
or equivalently, \(\frac{\hat{\theta} - \theta_0}{\text{se}(\hat{\theta})} \xrightarrow{d} N(0,1)\).
The three classical tests for \(H_0: \theta = \theta_0\) are:
All three converge to the same \(\chi^2_m\) distribution under \(H_0\) and have the same local asymptotic power.