Hypothesis Testing for ML

The null hypothesis $H_0$ is the default assumption you are trying to disprove. In ML model comparison:

$H_0: \mu_A = \mu_B$

where $\mu_A$ and $\mu_B$ are the true expected performances of models A and B. The null says there is no real difference between the models.

The alternative hypothesis $H_1$ is the claim tested against $H_0$; rejecting $H_0$ is evidence in favor of $H_1$.

• Two-sided: $H_1: \mu_A \neq \mu_B$ (the models differ)
• One-sided: $H_1: \mu_A > \mu_B$ (model A is better)

In ML, two-sided tests are more conservative and generally preferred unless you have a strong prior reason to test only one direction.

The p-value is the probability of observing a test statistic at least as extreme as the one computed from your data, assuming $H_0$ is true. The exact form depends on the alternative:

• One-sided ($H_1: \theta > \theta_0$): $p = P(T \geq t_{\text{obs}} \mid H_0)$
• Two-sided ($H_1: \theta \neq \theta_0$): $p = P(|T| \geq |t_{\text{obs}}| \mid H_0)$

Since two-sided tests are the default in ML model comparison, use the two-sided form unless you have committed in advance to a directional alternative. A small p-value means the observed data is unlikely under $H_0$, which is evidence against $H_0$.

The p-value is not the probability that $H_0$ is true. It is the probability of the data given $H_0$, not the probability of $H_0$ given the data.

The significance level $\alpha$ is the threshold for rejecting $H_0$. If and only if $p \leq \alpha$, we reject $H_0$ and declare the result "statistically significant." The standard choice is $\alpha = 0.05$, meaning we accept a 5% chance of falsely rejecting a true null hypothesis.

A $(1-\alpha)$ confidence interval for a parameter $\theta$ is a random interval $[L, U]$ whose coverage probability is at least $1-\alpha$:

$P_\theta(\theta \in [L, U]) \geq 1 - \alpha$

for every parameter value $\theta$ in the model. When equality holds, the interval has exact coverage.

For the difference in model performance $\delta = \mu_A - \mu_B$, a 95% CI provides a range of plausible values for the true difference. If the CI excludes zero, the difference is significant at $\alpha = 0.05$.

Confidence intervals are more informative than p-values alone because they communicate both statistical significance and effect size.

The False Discovery Rate is the expected proportion of false positives among all rejected hypotheses:

$\text{FDR} = \mathbb{E}\left[\frac{\text{false rejections}}{\text{total rejections}}\right]$

The Benjamini-Hochberg (BH) procedure controls FDR at level $q$:

1. Order the $m$ p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$
2. Find the largest $k$ such that $p_{(k)} \leq \frac{k}{m} \cdot q$
3. Reject all hypotheses with $p_{(i)} \leq p_{(k)}$

BH is less conservative than Bonferroni and is preferred when testing many hypotheses (e.g., comparing models on 50 datasets).

BH controls FDR at level $q$ under independence or positive regression dependence of the test statistics (Benjamini-Hochberg 1995; Benjamini-Yekutieli 2001). Under arbitrary dependence, use the Benjamini-Yekutieli correction, which replaces the threshold with $\frac{k}{m \cdot c(m)} q$ where $c(m) = \sum_{j=1}^m 1/j \approx \ln m$. This is strictly more conservative but valid without structural assumptions on the joint distribution of p-values.

Given $n$ test examples, let $d_i = \ell(A, x_i) - \ell(B, x_i)$ be the difference in loss between models A and B on example $i$. The paired t-test statistic is:

$t = \frac{\bar{d}}{s_d / \sqrt{n}}, \quad \bar{d} = \frac{1}{n}\sum_i d_i, \quad s_d^2 = \frac{1}{n-1}\sum_i (d_i - \bar{d})^2$

Under $H_0: \mathbb{E}[d_i] = 0$, $t$ follows a Student $t$-distribution with $n-1$ degrees of freedom exactly when the $d_i$ are i.i.d. Gaussian, and approximately for non-Gaussian $d_i$ by the CLT when $n$ is large. The $t$-distribution is robust to moderate departures from normality but breaks down for heavy-tailed or highly skewed $d_i$; in those cases use the Wilcoxon signed-rank test or a bootstrap.

A non-parametric alternative to the paired t-test. It does not assume normality of the differences $d_i$. Instead, it ranks the absolute differences $|d_i|$ and compares the sum of ranks for positive vs. negative differences. Use this when the differences are not approximately normal (e.g., heavy-tailed or skewed distributions).

The appropriate paired test when the per-example outcome is binary (correct/incorrect). Build a $2 \times 2$ contingency table of model A vs. model B outcomes on each test example and let $b$ be the count of examples where A is correct and B is wrong, $c$ the count where A is wrong and B is correct. Under $H_0$ that both models have the same error rate, the discordant pairs $b$ and $c$ are exchangeable, and

$\chi^2 = \frac{(b - c)^2}{b + c}$

is asymptotically $\chi^2_1$. For small counts ($b + c < 25$) use the exact binomial form: $P(X \geq \max(b,c))$ where $X \sim \text{Binomial}(b+c, 1/2)$. McNemar's test only uses discordant pairs. Concordant pairs (both right or both wrong) carry no signal about which model is better.

Use when $H_0$ is exchangeability of observations across groups (e.g., model A and model B perform identically on each example).

1. Compute the observed test statistic $T_{\text{obs}}$ (e.g., difference in accuracy).
2. For each of $B$ iterations, permute the group labels (swap which model produced each prediction) and recompute $T_b^*$ on the relabeled data.
3. The p-value is the fraction of permutation statistics at least as extreme: $p = \frac{1}{B}\sum_{b=1}^B \mathbf{1}[|T_b^*| \geq |T_{\text{obs}}|]$.

Permutation tests are exact under the exchangeability null and require no distributional assumptions.

Use to approximate the sampling distribution of a statistic when you cannot derive it analytically and the sample is too small for asymptotic approximations.

1. Compute the observed test statistic $T_{\text{obs}}$.
2. For each of $B$ iterations, resample the data with replacement (not permute), compute $T_b^*$, typically after recentering under $H_0$.
3. Construct confidence intervals from the empirical distribution of $T_b^*$, or compute a p-value from the recentered statistics.

Bootstrap tests estimate a sampling distribution; permutation tests construct a null distribution by exchange. They coincide asymptotically under some conditions but are not interchangeable in small samples.