Inner Product Spaces and Orthogonality

An inner product on a real vector space $V$ is a function $\langle \cdot, \cdot \rangle: V \times V \to \mathbb{R}$ satisfying:

1. Symmetry: $\langle u, v \rangle = \langle v, u \rangle$
2. Linearity in first argument: $\langle \alpha u + \beta v, w \rangle = \alpha \langle u, w \rangle + \beta \langle v, w \rangle$
3. Positive definiteness: $\langle v, v \rangle \geq 0$ with equality iff $v = 0$

The induced norm is $\|v\| = \sqrt{\langle v, v \rangle}$. The standard inner product on $\mathbb{R}^n$ is $\langle x, y \rangle = x^T y$.

Two vectors $u, v$ are orthogonal if and only if $\langle u, v \rangle = 0$, written $u \perp v$. A set $\{e_1, \ldots, e_k\}$ is orthonormal if and only if $\langle e_i, e_j \rangle = \delta_{ij}$ (Kronecker delta). The orthogonal complement of a subspace $W$ is $W^\perp = \{v \in V : \langle v, w \rangle = 0 \text{ for all } w \in W\}$.

The orthogonal projection of $v$ onto a subspace $W$ is the unique $\hat{v} \in W$ minimizing $\|v - w\|$ over $w \in W$, when this minimizer exists. If $\{e_1, \ldots, e_k\}$ is an orthonormal basis for $W$:

$\text{proj}_W(v) = \sum_{i=1}^{k} \langle v, e_i \rangle \, e_i$

The residual $v - \text{proj}_W(v)$ lies in $W^\perp$.

In finite dimensions every subspace admits a unique projection. In infinite-dimensional Hilbert spaces, existence and uniqueness of $\hat{v}$ require $W$ to be a closed subspace (Hilbert projection theorem). For a non-closed subspace, the infimum $\inf_{w \in W} \|v - w\|$ may not be attained in $W$; one then projects onto its closure $\overline{W}$.

Orthogonal projection is the geometry behind least squares. When the linear system $Ax = b$ has no exact solution, the closest reachable point is $\text{proj}_{\text{Col}(A)}(b) = Ax^+$ where $x^+ = A^+ b$ is given by the Moore-Penrose pseudoinverse. The residual $b - Ax^+$ lies in $\text{Col}(A)^\perp$, which is the optimality condition. The Pseudoinverse Geometry Lab makes this draggable: move $b$ and watch the residual stay perpendicular to $\text{Col}(A)$.

A Hilbert space is a complete inner product space: every Cauchy sequence converges. Finite-dimensional inner product spaces are automatically Hilbert spaces. Infinite-dimensional examples include $L^2$ (the space of square-integrable functions) and reproducing kernel Hilbert spaces (RKHS) used in kernel methods.

Let $\{e_1, e_2, \ldots\}$ be an orthonormal set in an inner product space $V$ (finite or countable). For every $v \in V$:

$\sum_i |\langle v, e_i \rangle|^2 \leq \|v\|^2$

with equality if and only if $v$ lies in the closure of $\text{span}\{e_i\}$. Each term $|\langle v, e_i \rangle|^2$ is the energy of $v$ along the direction $e_i$; the inequality says total projected energy cannot exceed total energy. Equality is Parseval's identity and characterizes complete orthonormal systems.

For real inner product spaces, the inner product is recoverable from the norm:

$\langle u, v \rangle = \tfrac{1}{4}\bigl(\|u + v\|^2 - \|u - v\|^2\bigr)$

Combined with the parallelogram law $\|u + v\|^2 + \|u - v\|^2 = 2\|u\|^2 + 2\|v\|^2$, this characterizes inner-product norms among all norms (Jordan-von Neumann theorem). Practically, it is how kernel methods compute inner products from distances: $\langle u, v \rangle = \tfrac{1}{2}(\|u\|^2 + \|v\|^2 - \|u - v\|^2)$, the trick that turns a distance kernel into an inner product kernel.

For $A, B \in \mathbb{R}^{m \times n}$:

$\langle A, B \rangle_F = \text{tr}(A^\top B) = \sum_{i,j} A_{ij} B_{ij}$

with induced norm $\|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2}$. Vectorizing $A$ and $B$ shows this is just the standard dot product on $\mathbb{R}^{mn}$. The Frobenius inner product gives the geometry behind low-rank matrix approximation: the Eckart-Young theorem says the best rank-$k$ approximation of $M$ in Frobenius norm is its truncated SVD, and the optimal value equals the residual singular-value energy.

For a symmetric positive-definite matrix $M \in \mathbb{R}^{d \times d}$:

$\langle x, y \rangle_M = x^\top M y$

with induced Mahalanobis distance $d_M(x, y) = \sqrt{(x - y)^\top M (x - y)}$. Choosing $M = \Sigma^{-1}$ (the inverse covariance) makes the geometry whitening-invariant: directions of high variance get down-weighted so that Mahalanobis distance reflects standardized deviation rather than raw Euclidean displacement. Quadratic discriminant analysis, Gaussian likelihoods, and metric learning (Xing et al. 2002, Weinberger and Saul 2009) all use Mahalanobis inner products. The factorization $M = L^\top L$ shows that $\langle x, y \rangle_M = \langle Lx, Ly \rangle$ - a Mahalanobis inner product is just a standard inner product after a linear transformation.