Asymptotic Statistics: M-Estimators, Delta Method, LAN

Let $X_n \xrightarrow{d} X$ in a metric space and let $g$ be measurable and continuous at every point of a set $C$ with $P(X \in C) = 1$. Then

$g(X_n) \xrightarrow{d} g(X).$

The same holds with $\xrightarrow{d}$ replaced by $\xrightarrow{p}$ or $\xrightarrow{a.s.}$.

Continuous functions preserve convergence. The "continuous on a set of $X$-measure one" condition allows $g$ to be discontinuous at points the limit never visits, which matters in practice (e.g., $g(x) = 1/x$ when $X$ is positive almost surely).

CMT is the most-used convergence result. Every "transform an estimator through a smooth function" argument starts here. Slutsky and the delta method are corollaries.

Discontinuity of $g$ at a point where the limit puts positive mass breaks the conclusion. Example: let $X_n = 1/n$ a.s., $X = 0$, and $g(x) = \mathbb{1}\{x = 0\}$. Then $X_n \xrightarrow{a.s.} 0$ but $g(X_n) = 0$ while $g(X) = 1$. The discontinuity at zero coincides with the limit's support.

If $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} c$ for a deterministic constant $c$, then jointly $(X_n, Y_n) \xrightarrow{d} (X, c)$, and therefore for any continuous $f$,

$f(X_n, Y_n) \xrightarrow{d} f(X, c).$

The standard corollaries:

$X_n + Y_n \xrightarrow{d} X + c, \quad X_n Y_n \xrightarrow{d} cX, \quad X_n / Y_n \xrightarrow{d} X/c \;\; (c \neq 0).$

A sequence converging in probability to a constant behaves like that constant in the limit. Marginal convergence $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{d} Y$ does not imply joint convergence in general; Slutsky works precisely because the $Y_n$ limit is degenerate (a point mass), so there is only one admissible coupling.

Slutsky is the glue that holds asymptotic arguments together. Every time you replace $\sigma$ with $\hat\sigma$ in a $z$-statistic and claim the limit is still standard normal, you are using Slutsky.

Slutsky fails if $Y_n$ has a non-degenerate limit. Example: if $X_n, Y_n$ are independent standard normals, $X_n \xrightarrow{d} N(0,1)$ and $Y_n \xrightarrow{d} N(0,1)$, but $X_n + Y_n$ has variance $2$, not the variance $1$ that "treating $Y_n$ as $0$" would suggest. The fix is joint convergence and CMT applied to the joint limit.

Univariate. If $\sqrt{n}(T_n - \mu) \xrightarrow{d} N(0, \sigma^2)$ and $g : \mathbb{R} \to \mathbb{R}$ is differentiable at $\mu$ with $g'(\mu) \neq 0$, then

$\sqrt{n}(g(T_n) - g(\mu)) \xrightarrow{d} N(0, [g'(\mu)]^2 \sigma^2).$

Multivariate. If $\sqrt{n}(T_n - \mu) \xrightarrow{d} N_k(0, \Sigma)$ and $g : \mathbb{R}^k \to \mathbb{R}^m$ has Jacobian $G(\mu) = \nabla g(\mu) \in \mathbb{R}^{m \times k}$ at $\mu$, then

$\sqrt{n}(g(T_n) - g(\mu)) \xrightarrow{d} N_m\!\left(0, G(\mu) \Sigma G(\mu)^\top\right).$

Second-order. If additionally $g'(\mu) = 0$ and $g''(\mu) \neq 0$, then

$n(g(T_n) - g(\mu)) \xrightarrow{d} \tfrac{1}{2} g''(\mu) \, \sigma^2 \chi^2_1.$

The convergence rate changes from $\sqrt{n}$ to $n$ and the limit is no longer normal.

A smooth function of an approximately normal random vector is itself approximately normal, with covariance pushed forward through the Jacobian. The second-order version covers the degenerate case where the gradient vanishes (e.g., the variance of the sample variance at the true variance has this structure).

The delta method is the workhorse of applied statistics. Variance of $\log(\hat p)$, of $1/\bar X$, of $\hat p_1 / \hat p_2$, of any smooth functional of moments. It is also the basis for variance-stabilizing transformations (the Anscombe and Freeman-Tukey families come from solving an ODE in $g'$).

Three failure modes. (1) $g'(\mu) = 0$: use the second-order version. (2) $g$ not differentiable at $\mu$: e.g., $g(x) = |x|$ at $0$ gives a folded-normal limit, not a normal. (3) $T_n$ converges at a non-$\sqrt{n}$ rate: the delta method must use that rate, not $\sqrt{n}$, in the expansion.

Under the stated regularity conditions, the Z-estimator $\hat\theta_n$ solving $\frac{1}{n}\sum_i \psi(X_i; \hat\theta_n) = 0$ satisfies

$\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} N\!\left(0, V(\theta_0)^{-1} A(\theta_0) V(\theta_0)^{-\top}\right),$

where $V(\theta_0) = \nabla_\theta \Psi(\theta_0) = E_P[\nabla_\theta \psi(X; \theta_0)]$ and $A(\theta_0) = E_P[\psi(X; \theta_0)\psi(X; \theta_0)^\top]$.

The variance is the sandwich form $V^{-1} A V^{-\top}$, with $V$ the "bread" and $A$ the "meat." When the model is correctly specified and $\psi$ is the score (MLE case), the information identity gives $V(\theta_0) = -A(\theta_0)$, the sandwich collapses to $A^{-1} = I(\theta_0)^{-1}$, and the variance equals the inverse Fisher information.

Expand the estimating equation around $\theta_0$:

$0 = \frac{1}{n}\sum_i \psi(X_i; \hat\theta_n) \approx \frac{1}{n}\sum_i \psi(X_i; \theta_0) + V(\theta_0) (\hat\theta_n - \theta_0).$

Solve for $\hat\theta_n - \theta_0$, multiply by $\sqrt{n}$:

$\sqrt{n}(\hat\theta_n - \theta_0) \approx -V(\theta_0)^{-1} \cdot \frac{1}{\sqrt{n}}\sum_i \psi(X_i; \theta_0).$

The right side is a CLT applied to mean-zero iid vectors with covariance $A(\theta_0)$, scaled by the deterministic matrix $-V(\theta_0)^{-1}$. The push-forward gives the sandwich variance.

This is the most general asymptotic-normality theorem in regular parametric inference. MLE asymptotic normality is the special case where the model is correctly specified and $\psi$ is the score. OLS, GMM, Huber's robust location, and quantile regression all follow as special cases with their own $V$ and $A$. The sandwich variance is also exactly what you should report when you suspect model misspecification: it is robust to $V \neq -A$, while the naive $A^{-1}$ "model-based" variance is not.

Three failure modes worth naming. (1) Lack of identification: if $V(\theta_0)$ is singular, the asymptotic variance is undefined and the estimator may converge at a slower rate (cube-root asymptotics for maximum-score estimators, $n^{1/3}$ rate). (2) Boundary parameter: if $\theta_0$ is on the boundary of $\Theta$ the limit is not normal but a half-normal or projection of a normal onto a cone (Self & Liang 1987, JASA). (3) Misspecification: $\Psi(\theta_0) = 0$ defines the "least-false" parameter, but the sandwich variance still applies; the naive Fisher-information-based variance does not.

Under the regularity conditions above, the maximum likelihood estimator $\hat\theta_n$ satisfies

$\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} N\!\left(0, I(\theta_0)^{-1}\right),$

where $I(\theta_0) = E_{\theta_0}[s(X; \theta_0) s(X; \theta_0)^\top] = -E_{\theta_0}[\nabla^2 \log f(X; \theta_0)]$ is the Fisher information matrix, with the second equality holding by the information identity.

The MLE is asymptotically normal centered at the truth with covariance $I(\theta_0)^{-1}/n$. This is the smallest variance any regular estimator can achieve (Cramér-Rao lower bound), so the MLE is asymptotically efficient. The result follows from the M-estimator theorem with $\psi = -s$, $V(\theta_0) = -E[\nabla s] = I(\theta_0)$, $A(\theta_0) = E[s s^\top] = I(\theta_0)$, and the sandwich $V^{-1} A V^{-\top}$ collapsing to $I^{-1} I I^{-1} = I^{-1}$.

Justifies the standard practice of reporting MLE point estimates with standard errors derived from the inverse observed information. It also underwrites likelihood-based inference: confidence intervals from inverting the Wald, score, or LRT statistic; profile-likelihood intervals for nuisance-parameter problems; and the asymptotic chi-squared distribution of $-2 \log \Lambda$ in nested-model testing.

Regularity fails at boundary parameters (e.g., $\sigma = 0$ in $N(\mu, \sigma^2)$), non-identified models (mixture models with unknown component count), models where the support depends on the parameter (e.g., $\text{Uniform}(0, \theta)$, where $\hat\theta_{\text{MLE}} = X_{(n)}$ converges at rate $n$ to an exponential), and singular information ($I(\theta_0)$ not full rank, e.g., at a point of non-identification in mixture models).

Under DQM at $\theta_0$, the log-likelihood ratio admits the LAN expansion

$\log \frac{dP_{\theta_0 + h/\sqrt{n}}^{\otimes n}}{dP_{\theta_0}^{\otimes n}}(X_1, \ldots, X_n) = h^\top \Delta_n - \frac{1}{2} h^\top I(\theta_0) h + o_{P_{\theta_0}}(1),$

where the central sequence $\Delta_n = \frac{1}{\sqrt{n}}\sum_i s(X_i; \theta_0)$ satisfies $\Delta_n \xrightarrow{d} N_k(0, I(\theta_0))$ under $P_{\theta_0}$.

Le Cam's three lemmas then give:

1. The product measures $P_{\theta_0 + h/\sqrt{n}}^{\otimes n}$ and $P_{\theta_0}^{\otimes n}$ are mutually contiguous.
2. Convergence in distribution under $P_{\theta_0}$ transfers to $P_{\theta_0 + h/\sqrt{n}}$ via the change-of-measure formula.
3. The asymptotic experiment is the Gaussian shift experiment $\Delta \sim N_k(I(\theta_0) h, I(\theta_0))$: every regular procedure in the original problem has a corresponding procedure in the Gaussian experiment with the same asymptotic risk.

At the local scale $h/\sqrt{n}$ around the truth, every regular parametric problem reduces to a Gaussian shift problem. The sufficient statistic is the central sequence $\Delta_n$ (the normalized score), the parameter is the local direction $h$, and the Fisher information sets the signal-to-noise ratio. This is the deepest structural result in classical parametric statistics: it says the MLE is asymptotically optimal not because of clever proofs but because it is the natural estimator in the limiting Gaussian experiment.

Differentiability in quadratic mean (DQM) is weaker than pointwise differentiability of $\log f$: it requires only that the square-root density $\sqrt{f}$ be differentiable in $L^2$ as a function of $\theta$. This handles models with bounded support, non-smooth densities, and many other "almost regular" cases where the textbook regularity conditions fail.

LAN underwrites the asymptotic minimax lower bound: no regular estimator can have asymptotic variance smaller than $I(\theta_0)^{-1}$. It also gives the asymptotic equivalence of Wald, score, and LRT tests under the null and under contiguous alternatives, and it is the foundation of semiparametric efficiency theory (Bickel-Klaassen-Ritov-Wellner 1993): in semiparametric models the analogous expansion identifies the efficient score as the projection of the parametric score onto the orthogonal complement of the nuisance tangent space.

LAN fails for non-regular models. Canonical examples include parameter-dependent support (e.g., Uniform$(0, \theta)$, where the MLE converges at rate $n$ rather than $\sqrt{n}$ and the limit is exponential), boundary parameter problems (parameter on the edge of $\Theta$, where the limit is a censored normal), change-point models (where the rate is $n$ and the limit is a compound Poisson functional), mixture models at non-identified configurations, and unidentified parameters generally. Note that the standard Cauchy location family $f_\theta(x) = 1/[\pi(1 + (x-\theta)^2)]$ is regular: its Fisher information equals $1/2$ and LAN holds, so the location MLE is regular and asymptotically normal at the $\sqrt{n}$ rate. The Cauchy sample mean is pathological because the Cauchy has no first moment, but likelihood-based location inference is fine. Many machine-learning "models" with high-dimensional parameters whose interior is unclear also fall outside LAN. The corresponding theory is LAMN (locally asymptotically mixed normal) and LAQ (locally asymptotically quadratic), with weaker optimality conclusions.

Under the null $H_0 : \theta = \theta_0$ and LAN, define the three test statistics:

• Wald. $W_n = n (\hat\theta_n - \theta_0)^\top I(\theta_0) (\hat\theta_n - \theta_0)$.
• Score (Rao / Lagrange multiplier). $R_n = \frac{1}{n} \nabla \ell_n(\theta_0)^\top I(\theta_0)^{-1} \nabla \ell_n(\theta_0) = \Delta_n^\top I(\theta_0)^{-1} \Delta_n$.
• Likelihood ratio. $L_n = -2 \log \frac{L_n(\theta_0)}{L_n(\hat\theta_n)} = 2(\ell_n(\hat\theta_n) - \ell_n(\theta_0))$.

Then under $H_0$, all three converge to the same chi-squared limit:

$W_n, R_n, L_n \xrightarrow{d} \chi^2_k \quad \text{where} \;\; k = \dim(\theta).$

Moreover, $W_n - L_n = o_p(1)$ and $R_n - L_n = o_p(1)$, so the three tests are asymptotically equivalent under $H_0$ and under contiguous alternatives $\theta_n = \theta_0 + h/\sqrt{n}$ (where the common limit becomes a non-central $\chi^2_k(\lambda)$ with non-centrality $\lambda = h^\top I(\theta_0) h$).

The three tests measure "distance from $H_0$" in three different but asymptotically equivalent ways. Wald measures distance in the parameter space (how far is $\hat\theta_n$ from $\theta_0$, weighted by the information). Score measures the slope of the log-likelihood at $\theta_0$ (is the gradient $\Delta_n$ near zero, or far?). LRT measures the depth of the log-likelihood (how much does the maximum exceed the null value?). All three reduce to the same quadratic form in the central sequence $\Delta_n$ under LAN, and the chi-squared limit is the distribution of the squared length of a standard normal $k$-vector.

Justifies reporting any of the three test statistics in practice with the same chi-squared reference distribution. Picks among them are finite-sample concerns: Wald requires the MLE and an estimate of $I(\theta_0)$; score does not require fitting the alternative model and is preferred in some computational settings; LRT is invariant under parameter transformations and tends to have better small-sample behavior in nested-model comparisons. The non-central chi-squared limit under contiguous alternatives gives the local power calculation: power $\to P(\chi^2_k(\lambda) > \chi^2_{k, 1-\alpha})$, a function of the non-centrality $\lambda = h^\top I(\theta_0) h$.

The three tests can disagree substantially in finite samples, particularly when $n / k$ is small or the model is misspecified. Wald is parameterization-dependent (Wald CIs for $\theta$ and for $g(\theta)$ are not invariant under reparameterization, while LRT CIs are). Under misspecification only the robust score test with a sandwich denominator preserves a valid chi-squared limit; the naive Wald and LRT statistics use the wrong variance. Under boundary conditions the limit is a mixture of chi-squared distributions, not a single $\chi^2$ (Self-Liang 1987).