Orthogonal Projection

The Best Approximation Theorem

Let $V$ be a real inner-product space and let $S\subseteq V$ be a finite-dimensional subspace. For every $f\in V$ , there is a unique vector $g\in S$ such that

\|f-g\|=\min_{\varphi\in S}\|f-\varphi\|

The vector $g$ is the orthogonal projection of $f$ onto $S$ , written $g=P_Sf$ . It is characterized by two conditions:

g\in S

Equivalently,

\begin{aligned} &\langle f-g,\varphi\rangle=0 \\ &\text{for every }\varphi\in S \end{aligned}

In a Hilbert space the same theorem holds for $every closed subspace$ . A finite-dimensional subspace is automatically closed.

Why Orthogonality Gives the Minimum

Let $g=P_Sf$ and choose any $\varphi\in S$ . Since $g-\varphi\in S$ and $f-g\perp S$ , the $Pythagorean identity$ gives

f-\varphi=(f-g)+(g-\varphi)

Equality holds only when $\varphi=g$ , which proves both optimality and uniqueness.

Formula for an Orthonormal Basis

If $\varphi_1,\ldots,\varphi_n$ is an orthonormal basis of $S$ , then

P_Sf=\sum_{i=1}^n\langle f,\varphi_i\rangle\varphi_i

The second line follows by splitting $f$ into the $perpendicular components$ $P_Sf$ and $f-P_Sf$ .

For a real matrix $Q\in\mathbb{R}^{m\times n}$ with orthonormal columns,

P=QQ^T

The matrix $P$ projects vectors in $\mathbb{R}^m$ onto $\operatorname{col}(Q)$ .

Projecting a Vector onto a One Dimensional Subspace

The two vectors are

f=\begin{pmatrix}1\\3\end{pmatrix}

Project the vector $f$ onto $S=\operatorname{span}\{u\}$ . The $projection coefficient$ is

\frac{\langle f,u\rangle}{\langle u,u\rangle}=\frac{1\cdot2+3\cdot1}{2^2+1^2}=1

The calculation gives:

g=P_Sf=u=\begin{pmatrix}2\\1\end{pmatrix}

Projection of

f

onto

\operatorname{span}\{u\}

The cyan vector is

f

, the orange vector is

g=P_Sf

, and the purple segment is the perpendicular residual

r=f-g

$x$: $[0, 2]$
$y$: $[0, 3]$

Construction from an Arbitrary Basis

Let $\psi_1,\ldots,\psi_n$ be any basis of $S$ and write

g=\sum_{i=1}^n\alpha_i\psi_i

The orthogonality conditions give the $Gram system$

\begin{aligned} &\sum_{i=1}^n\langle\psi_i,\psi_k\rangle\alpha_i=\langle f,\psi_k\rangle, \\ &k=1,\ldots,n \end{aligned}

In this system, $G_{ki}=\langle\psi_i,\psi_k\rangle$ and $c_k=\langle f,\psi_k\rangle$ . Since the basis vectors are linearly independent,

\begin{aligned} &z^TGz=\left\|\sum_{i=1}^nz_i\psi_i\right\|^2>0 \\ &\text{for }z\ne0 \end{aligned}

The inequality above shows that $G$ is symmetric positive definite, so the projection coefficients are unique.

If the columns of a full-column-rank matrix $A$ are the basis vectors, then

P=A(A^TA)^{-1}A^T

The matrix $P$ is the same orthogonal projector. This formula gives the algebraic form of the projector. For numerical computation, obtain the projection with $\mathrm{QR}$ or $\mathrm{SVD}$ because forming $A^TA$ squares the condition number.

Function Space Example

Project $f(x)=x^2$ onto $S=\operatorname{span}\{1,x\}$ using

\langle f,g\rangle=\int_{-1}^{1}f(x)g(x)\,dx

The functions $1$ and $x$ are orthogonal. Their coefficients are

c_0=\frac{\langle x^2,1\rangle}{\langle1,1\rangle}=\frac{2/3}{2}=\frac13

The calculation gives:

P_Sf(x)=\frac13

The linear term vanishes because $x^3$ is $odd on the symmetric interval$ .

Properties of an Orthogonal Projector

Property	Meaning
$P^2=P$	projecting twice changes nothing
$P^T=P$	the range and null space are orthogonal
$\operatorname{range}(P)=S$	every output lies in the target subspace
$\ker(P)=S^\perp$	perpendicular components are removed

Idempotence alone is not enough. An oblique projector also satisfies $P^2=P$ and may be asymmetric. $Symmetry distinguishes the Euclidean orthogonal projector$ .

The Best Approximation Theorem

Let $V$ be a real inner-product space and let $S\subseteq V$ be a finite-dimensional subspace. For every $f\in V$ , there is a unique vector $g\in S$ such that

\|f-g\|=\min_{\varphi\in S}\|f-\varphi\|

The vector $g$ is the orthogonal projection of $f$ onto $S$ , written $g=P_Sf$ . It is characterized by two conditions:

g\in S

Equivalently,

\begin{aligned} &\langle f-g,\varphi\rangle=0 \\ &\text{for every }\varphi\in S \end{aligned}

In a Hilbert space the same theorem holds for $every closed subspace$ . A finite-dimensional subspace is automatically closed.

Why Orthogonality Gives the Minimum

Let $g=P_Sf$ and choose any $\varphi\in S$ . Since $g-\varphi\in S$ and $f-g\perp S$ , the $Pythagorean identity$ gives

f-\varphi=(f-g)+(g-\varphi)

Equality holds only when $\varphi=g$ , which proves both optimality and uniqueness.

Formula for an Orthonormal Basis

If $\varphi_1,\ldots,\varphi_n$ is an orthonormal basis of $S$ , then

P_Sf=\sum_{i=1}^n\langle f,\varphi_i\rangle\varphi_i

The second line follows by splitting $f$ into the $perpendicular components$ $P_Sf$ and $f-P_Sf$ .

For a real matrix $Q\in\mathbb{R}^{m\times n}$ with orthonormal columns,

P=QQ^T

The matrix $P$ projects vectors in $\mathbb{R}^m$ onto $\operatorname{col}(Q)$ .

Projecting a Vector onto a One Dimensional Subspace

The two vectors are

f=\begin{pmatrix}1\\3\end{pmatrix}

Project the vector $f$ onto $S=\operatorname{span}\{u\}$ . The $projection coefficient$ is

\frac{\langle f,u\rangle}{\langle u,u\rangle}=\frac{1\cdot2+3\cdot1}{2^2+1^2}=1

The calculation gives:

g=P_Sf=u=\begin{pmatrix}2\\1\end{pmatrix}

Projection of

f

onto

\operatorname{span}\{u\}

The cyan vector is

f

, the orange vector is

g=P_Sf

, and the purple segment is the perpendicular residual

r=f-g

$x$: $[0, 2]$
$y$: $[0, 3]$

Construction from an Arbitrary Basis

Let $\psi_1,\ldots,\psi_n$ be any basis of $S$ and write

g=\sum_{i=1}^n\alpha_i\psi_i

The orthogonality conditions give the $Gram system$

\begin{aligned} &\sum_{i=1}^n\langle\psi_i,\psi_k\rangle\alpha_i=\langle f,\psi_k\rangle, \\ &k=1,\ldots,n \end{aligned}

In this system, $G_{ki}=\langle\psi_i,\psi_k\rangle$ and $c_k=\langle f,\psi_k\rangle$ . Since the basis vectors are linearly independent,

\begin{aligned} &z^TGz=\left\|\sum_{i=1}^nz_i\psi_i\right\|^2>0 \\ &\text{for }z\ne0 \end{aligned}

The inequality above shows that $G$ is symmetric positive definite, so the projection coefficients are unique.

If the columns of a full-column-rank matrix $A$ are the basis vectors, then

P=A(A^TA)^{-1}A^T

Function Space Example

Project $f(x)=x^2$ onto $S=\operatorname{span}\{1,x\}$ using

\langle f,g\rangle=\int_{-1}^{1}f(x)g(x)\,dx

The functions $1$ and $x$ are orthogonal. Their coefficients are

c_0=\frac{\langle x^2,1\rangle}{\langle1,1\rangle}=\frac{2/3}{2}=\frac13

The calculation gives:

P_Sf(x)=\frac13

The linear term vanishes because $x^3$ is $odd on the symmetric interval$ .

Properties of an Orthogonal Projector

Property	Meaning
$P^2=P$	projecting twice changes nothing
$P^T=P$	the range and null space are orthogonal
$\operatorname{range}(P)=S$	every output lies in the target subspace
$\ker(P)=S^\perp$	perpendicular components are removed

Idempotence alone is not enough. An oblique projector also satisfies $P^2=P$ and may be asymmetric. $Symmetry distinguishes the Euclidean orthogonal projector$ .