Common Probability Distributions

A single binary trial with success probability $p \in [0, 1]$.

PMF: $P(X = k) = p^k (1-p)^{1-k}$ for $k \in \{0, 1\}$

Mean: $\mathbb{E}[X] = p$

Variance: $\text{Var}(X) = p(1 - p)$, maximized at $p = 1/2$ with value $1/4$.

MGF: $\mathbb{E}[e^{tX}] = 1 - p + p e^t$ for all $t \in \mathbb{R}$.

When it arises: Binary classification labels, coin flips, any yes/no outcome. The log-likelihood of Bernoulli data gives rise to the cross-entropy loss used in logistic regression. The Bernoulli MGF is the starting point of the Chernoff bound: applying Markov's inequality to $e^{tS_n}$ for a sum of independent Bernoullis gives the canonical exponential tail bound on a binomial sum.

The number of successes in $n$ independent Bernoulli$(p)$ trials.

PMF: $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k = 0, 1, \ldots, n$

Mean: $\mathbb{E}[X] = np$

Variance: $\text{Var}(X) = np(1 - p)$

MGF: $\mathbb{E}[e^{tX}] = (1 - p + p e^t)^n$ for all $t \in \mathbb{R}$ (the $n$-th power of the Bernoulli MGF).

Key facts:

• If $X_1, \ldots, X_n \sim \text{Bern}(p)$ independently, then $\sum_i X_i \sim \text{Bin}(n, p)$.
• As $n \to \infty$ with $np \to \lambda$, the Binomial converges to $\text{Poisson}(\lambda)$.
• For fixed $p$, the standardized sum $(X - np)/\sqrt{np(1-p)} \Rightarrow \mathcal{N}(0, 1)$ by the central limit theorem. The error is $O(1/\sqrt{np(1-p)})$ by Berry-Esseen.

When it arises: Empirical risk on $n$ samples with 0-1 loss is $\text{Bin}(n, p)/n$ where $p$ is the true error rate. Slud's inequality lower-bounds binomial tails and is used in VC sample-complexity lower bounds.

A single trial with $K$ possible outcomes, where outcome $k$ has probability $p_k \geq 0$ and $\sum_k p_k = 1$. Generalizes Bernoulli (recovered when $K = 2$).

PMF: $P(X = k) = p_k$ for $k \in \{1, \ldots, K\}$.

Mean of the one-hot encoding $\mathbf{e}_X \in \{0,1\}^K$: $\mathbb{E}[\mathbf{e}_X] = (p_1, \ldots, p_K)$.

When it arises: Multi-class classification labels. The softmax output of a neural network parameterizes a Categorical: $p_k = e^{z_k}/\sum_j e^{z_j}$ where $z = Wx + b$ are the logits. Cross-entropy loss is $-\log p_y$ where $y$ is the true class — this is the Categorical negative log-likelihood. Language models predict a Categorical over vocabulary tokens at every position. A Multinomial is the sum of $n$ independent Categoricals with the same parameter vector.

Models the count of events in a fixed interval when events occur independently at a constant rate $\lambda > 0$.

PMF: $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$

Mean: $\mathbb{E}[X] = \lambda$

Variance: $\text{Var}(X) = \lambda$

MGF: $\mathbb{E}[e^{tX}] = \exp(\lambda(e^t - 1))$ for all $t \in \mathbb{R}$.

When it arises: Count data: number of clicks, arrivals, mutations. The mean equals the variance; if your count data has variance much larger than the mean, the Poisson model is wrong (use negative binomial instead). The Poisson MGF gives sub-exponential tail bounds; for large deviations, use the Bennett or Bernstein inequality rather than a Gaussian approximation.

The number of trials until the first success in independent Bernoulli$(p)$ trials. (Convention: $X$ includes the successful trial.)

PMF: $P(X = k) = (1-p)^{k-1} p$ for $k = 1, 2, 3, \ldots$

Mean: $\mathbb{E}[X] = 1/p$

Variance: $\text{Var}(X) = (1-p)/p^2$

Key property: The Geometric distribution is the only discrete distribution with the memoryless property: $P(X > s + t \mid X > s) = P(X > t)$.

When it arises in ML: Coupon-collector arguments (the expected time to see all $K$ classes from a uniform stream is $K H_K \approx K \log K$); PAC sample-complexity lower bounds; the geometric series bounds in TD-learning convergence.

The number of trials until the $r$-th success in independent Bernoulli$(p)$ trials. Generalizes the Geometric (recovered when $r = 1$).

PMF: $P(X = k) = \binom{k-1}{r-1} p^r (1-p)^{k-r}$ for $k = r, r+1, \ldots$

Mean: $\mathbb{E}[X] = r/p$

Variance: $\text{Var}(X) = r(1-p)/p^2$

Key fact: The variance exceeds the mean by a factor of $1/p$, so the negative binomial is a natural overdispersed alternative to the Poisson. Sums of $r$ independent $\text{Geom}(p)$ variables are $\text{NB}(r, p)$.

When it arises: Count data with overdispersion (variance $>$ mean), where the Poisson assumption fails. Common in NLP word-frequency models and biological sequencing data.

The multivariate generalization of the Binomial. In $n$ trials, each of $K$ categories occurs with probability $p_k$, and $X_k$ counts category $k$.

PMF: $P(X_1 = x_1, \ldots, X_K = x_K) = \frac{n!}{x_1! \cdots x_K!} p_1^{x_1} \cdots p_K^{x_K}$

where $\sum_k x_k = n$ and $\sum_k p_k = 1$.

Marginals: Each $X_k \sim \text{Bin}(n, p_k)$.

Relationship: A Multinomial is the sum of $n$ i.i.d. Categorical$(p_1, \ldots, p_K)$ trials. The single-trial case ($n = 1$) is the Categorical distribution defined above; this is what a softmax layer parameterizes.

When it arises: Bag-of-words models, topic models, document language models with $n$-token contexts. Empirical class frequencies in a multi-class classifier with $n$ examples are $\text{Mult}(n, p)/n$.

Constant density on the interval $[a, b]$.

PDF: $f(x) = \frac{1}{b - a}$ for $x \in [a, b]$, zero otherwise.

Mean: $\mathbb{E}[X] = \frac{a + b}{2}$

Variance: $\text{Var}(X) = \frac{(b - a)^2}{12}$

When it arises: Maximum entropy distribution on a bounded interval with no other constraints. Used in random initialization, random search, and as the base distribution for inverse transform sampling.

The most important continuous distribution. Parameterized by mean $\mu$ and variance $\sigma^2 > 0$.

PDF: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$

Mean: $\mathbb{E}[X] = \mu$

Variance: $\text{Var}(X) = \sigma^2$

MGF: $\mathbb{E}[e^{tX}] = \exp(\mu t + \sigma^2 t^2 / 2)$ for all $t \in \mathbb{R}$.

Standard normal moments (for $Z \sim \mathcal{N}(0,1)$): $\mathbb{E}[Z^2] = 1$, $\mathbb{E}[Z^4] = 3$, and in general $\mathbb{E}[Z^{2k}] = (2k-1)!! = (2k)!/(2^k k!)$ for $k \geq 0$ (odd moments vanish by symmetry). The fact that $\mathbb{E}[Z^4] = 3$ is what makes $\text{Var}(Z^2) = \mathbb{E}[Z^4] - 1 = 2$, the variance of a $\chi^2(1)$ summand.

Tail bound (Mills ratio): For $Z \sim \mathcal{N}(0, 1)$ and $t > 0$, $\frac{1}{t + 1/t} \cdot \frac{e^{-t^2/2}}{\sqrt{2\pi}} \leq \mathbb{P}(Z > t) \leq \frac{1}{t} \cdot \frac{e^{-t^2/2}}{\sqrt{2\pi}}.$ The simpler bound $\mathbb{P}(Z > t) \leq e^{-t^2/2}$ for $t > 0$ follows from the Chernoff method on the Gaussian MGF and is what underlies sub-Gaussian concentration. The polynomial factor $1/t$ makes the Gaussian tail tighter than the bare sub-Gaussian bound by a $1/t$ factor — relevant in high-dimensional probability and union-bound calculations.

The normalizing constant $1/\sqrt{2\pi\sigma^2}$ ensures $\int_{-\infty}^{\infty} f(x)\,dx = 1$. This requires the Gaussian integral $\int_{-\infty}^{\infty} e^{-x^2/2}\,dx = \sqrt{2\pi}$.

For $X \in \mathbb{R}^d$ with mean vector $\mu \in \mathbb{R}^d$ and positive definite covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$:

PDF: $f(x) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \exp\!\left(-\frac{1}{2}(x - \mu)^\top \Sigma^{-1} (x - \mu)\right)$

Key properties:

• Marginals are Gaussian: if $X \sim \mathcal{N}(\mu, \Sigma)$, any subset of coordinates is also multivariate Gaussian.
• For jointly Gaussian distributions, uncorrelated implies independent. Marginal Gaussianity alone does not suffice. A standard counterexample: $X \sim \mathcal{N}(0, 1)$ and $Y = X \cdot \mathbb{1}\{|X| \leq c\} - X \cdot \mathbb{1}\{|X| > c\}$. Both marginals are Gaussian and the pair is uncorrelated for an appropriate $c$, yet $X$ and $Y$ are dependent (since $|X| = |Y|$).
• Affine transformations preserve Gaussianity: $AX + b \sim \mathcal{N}(A\mu + b, A\Sigma A^\top)$.

When it arises: Central limit theorem, Bayesian linear regression posterior, Gaussian processes, VAE latent spaces.

Models waiting times between events in a Poisson process with rate $\lambda > 0$.

PDF: $f(x) = \lambda e^{-\lambda x}$ for $x \geq 0$

Mean: $\mathbb{E}[X] = 1/\lambda$

Variance: $\text{Var}(X) = 1/\lambda^2$

Key property: Memoryless: $P(X > s + t \mid X > s) = P(X > t)$. This is the only continuous distribution with this property.

Relationship: $\text{Exp}(\lambda) = \text{Gamma}(1, \lambda)$.

Generalizes the Exponential. Shape $\alpha > 0$, rate $\beta > 0$.

PDF: $f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x}$ for $x > 0$

where $\Gamma(\alpha) = \int_0^\infty t^{\alpha-1} e^{-t}\,dt$ is the Gamma function.

Mean: $\mathbb{E}[X] = \alpha/\beta$

Variance: $\text{Var}(X) = \alpha/\beta^2$

Special cases:

• $\text{Gamma}(1, \lambda) = \text{Exp}(\lambda)$
• $\text{Gamma}(n/2, 1/2) = \chi^2(n)$
• Sum of $n$ independent $\text{Exp}(\beta)$ variables is $\text{Gamma}(n, \beta)$

When it arises: Conjugate prior for the Poisson rate and the precision (inverse variance) of a Gaussian. Models positive quantities with skew.

Defined on $[0, 1]$ with shape parameters $\alpha, \beta > 0$.

PDF: $f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}$ for $x \in [0, 1]$

where $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}$.

Mean: $\mathbb{E}[X] = \frac{\alpha}{\alpha + \beta}$

Variance: $\text{Var}(X) = \frac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}$

Key fact: The Beta is the conjugate prior for the Bernoulli/Binomial parameter $p$. If $p \sim \text{Beta}(\alpha, \beta)$ and $X \mid p \sim \text{Bin}(n, p)$ with $k$ successes, then $p \mid X \sim \text{Beta}(\alpha + k, \beta + n - k)$.

Special cases: $\text{Beta}(1, 1) = \text{Unif}(0, 1)$.

The sum of $k$ independent squared standard normals. Degrees of freedom $k$.

PDF: $f(x) = \frac{x^{k/2 - 1} e^{-x/2}}{2^{k/2}\Gamma(k/2)}$ for $x > 0$

Mean: $\mathbb{E}[X] = k$

Variance: $\text{Var}(X) = 2k$

MGF: $\mathbb{E}[e^{\lambda X}] = (1 - 2\lambda)^{-k/2}$ for $\lambda < 1/2$, with $\mathbb{E}[e^{\lambda X}] = +\infty$ for $\lambda \geq 1/2$. This is the half-line MGF that drives every $\chi^2$ tail bound. Applying the Chernoff method and optimizing over $\lambda \in (0, 1/2)$ gives the standard concentration result: for $X \sim \chi^2(k)$ and $t \geq 0$, $\mathbb{P}(X \geq k + t) \leq \exp\!\left(-\tfrac{t^2}{4(k + t)}\right),$ which sharpens to the Laurent-Massart bound $\mathbb{P}(X - k \geq 2\sqrt{kt} + 2t) \leq e^{-t}$. These bounds are what appear in the random-projection lemma (B.12 in Shai & Shai), the Johnson-Lindenstrauss lemma, and PCA perturbation analyses.

Relationship: $\chi^2(k) = \text{Gamma}(k/2, 1/2)$.

If $Z_1, \ldots, Z_k \sim \mathcal{N}(0,1)$ independently, then $\sum_i Z_i^2 \sim \chi^2(k)$.

When it arises: Hypothesis testing (goodness-of-fit, likelihood ratio tests), analysis of variance. The sample variance of Gaussian data is proportional to a $\chi^2$. Important in concentration: the $\chi^2$ is sub-exponential but not sub-Gaussian, because the MGF blows up at $\lambda = 1/2$ rather than being defined on all of $\mathbb{R}$.

Arises when estimating the mean of a Gaussian population with unknown variance. Degrees of freedom $\nu > 0$.

PDF: $f(x) = \frac{\Gamma((\nu+1)/2)}{\sqrt{\nu\pi}\,\Gamma(\nu/2)}\left(1 + \frac{x^2}{\nu}\right)^{-(\nu+1)/2}$

Mean: $\mathbb{E}[X] = 0$ for $\nu > 1$ (undefined for $\nu \leq 1$)

Variance: $\text{Var}(X) = \frac{\nu}{\nu - 2}$ for $\nu > 2$

Construction: If $Z \sim \mathcal{N}(0,1)$ and $V \sim \chi^2(\nu)$ are independent, then $T = Z / \sqrt{V/\nu} \sim t(\nu)$.

Key fact: Heavier tails than the Gaussian. As $\nu \to \infty$, $t(\nu) \to \mathcal{N}(0, 1)$. For small $\nu$, the heavy tails accommodate outliers. This is why the Student-t is used in robust statistics.

The ratio of two independent scaled chi-squared variables.

Construction: If $U \sim \chi^2(d_1)$ and $V \sim \chi^2(d_2)$ are independent, then $F = \frac{U/d_1}{V/d_2} \sim F(d_1, d_2)$.

Mean: $\mathbb{E}[X] = \frac{d_2}{d_2 - 2}$ for $d_2 > 2$

When it arises: ANOVA, comparing two model fits (F-test for nested models), testing whether a group of regression coefficients is jointly zero.

Relationship: If $T \sim t(\nu)$, then $T^2 \sim F(1, \nu)$.

Location $x_0$, scale $\gamma > 0$.

PDF: $f(x) = \frac{1}{\pi\gamma\left[1 + \left(\frac{x - x_0}{\gamma}\right)^2\right]}$

Mean: Does not exist (the integral diverges).

Variance: Does not exist.

Key fact: The Cauchy is $t(1)$. The Student-t with one degree of freedom. It is the standard example of a distribution with no mean. The sample mean of $n$ i.i.d. Cauchy variables has the same distribution as a single observation. The law of large numbers does not apply. This is why finite moments matter for concentration inequalities.

The multivariate generalization of the Beta, defined on the $(K-1)$-simplex $\{x \in \mathbb{R}^K : x_k \geq 0, \sum_k x_k = 1\}$.

PDF: $f(x_1, \ldots, x_K) = \frac{\Gamma(\sum_k \alpha_k)}{\prod_k \Gamma(\alpha_k)} \prod_{k=1}^K x_k^{\alpha_k - 1}$

Marginals: Each $X_k \sim \text{Beta}(\alpha_k, \sum_{j \neq k} \alpha_j)$.

Mean: $\mathbb{E}[X_k] = \alpha_k / \sum_j \alpha_j$

Key fact: Conjugate prior for the Multinomial/Categorical. If $\theta \sim \text{Dir}(\alpha)$ and $X \mid \theta \sim \text{Mult}(n, \theta)$ with counts $c_1, \ldots, c_K$, then $\theta \mid X \sim \text{Dir}(\alpha_1 + c_1, \ldots, \alpha_K + c_K)$.

When it arises: Latent Dirichlet Allocation (LDA), Bayesian multi-class models, any model requiring a prior over probability vectors.