Inner Products

Using Inner Products to Describe Geometry

A vector space has operations for addition and scalar multiplication. An inner product determines vector lengths, angles between vectors, and whether two vectors are orthogonal. For complex vectors, the inner product used below is $conjugate-linear in its first argument$ and linear in its second.

For $x,y,z\in\mathbb{C}^n$ and $\lambda,\mu\in\mathbb{C}$ , an inner product satisfies

\langle \lambda x+\mu y,z\rangle=\overline\lambda\langle x,z\rangle+\overline\mu\langle y,z\rangle

Conjugate symmetry guarantees that $\langle x,x\rangle$ is real. Over $\mathbb{R}^n$ , complex conjugation disappears, so the inner product is symmetric and bilinear.

Matrix Weighted Inner Products

For real vectors, a matrix weighted inner product is written as:

\langle x,y\rangle_A=x^TAy

This formula defines an inner product exactly when $A$ is $symmetric positive definite$ . Symmetry of $A$ makes the result symmetric in $x$ and $y$ , while $x^TAx>0$ for every nonzero $x$ gives strict positivity.

For complex vectors, the corresponding formula is

\langle x,y\rangle_A=x^HAy

In this formula, $A$ must be Hermitian positive definite. Merely saying that $A$ has positive entries or positive diagonal entries is not enough.

The standard Euclidean inner products are recovered with $A=I$ :

\langle x,y\rangle=x^Ty\qquad\text{over }\mathbb{R}^n

Vector Lengths with Angles and Orthogonality

Every inner product defines the following norm:

\|x\|_A=\sqrt{\langle x,x\rangle_A}

This norm satisfies the Cauchy-Schwarz inequality:

|\langle x,y\rangle_A|\le\|x\|_A\|y\|_A

For nonzero real vectors, this inequality ensures that the angle can be defined by

\cos\theta_A=\frac{\langle x,y\rangle_A}{\|x\|_A\|y\|_A}

Vectors are $A$ -orthogonal when $\langle x,y\rangle_A=0$ . Changing $A$ changes the geometry, so vectors that are perpendicular in the Euclidean metric $need not be perpendicular in a weighted metric$ .

Comparing Euclidean and Weighted Angles

Let

\begin{aligned} &A=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \\ &x=\begin{pmatrix}1\\0\end{pmatrix}, \\ &y=\begin{pmatrix}0\\1\end{pmatrix} \end{aligned}

The positive eigenvalues confirm that $A$ is symmetric positive definite. Although $x^Ty=0$ , the $weighted geometry gives$

\langle x,y\rangle_A=x^TAy=1

The same coordinate vectors are perpendicular in the Euclidean geometry but form a $60^\circ$ angle in the $A$ -geometry.

Orthogonal Complex Vectors with Conjugation

For the standard inner product, take

x=\begin{pmatrix}1\\i\end{pmatrix},\qquad y=\begin{pmatrix}i\\1\end{pmatrix}

Then

\begin{aligned} \langle x,y\rangle &=x^Hy \\ &=\begin{pmatrix}1&-i\end{pmatrix}\begin{pmatrix}i\\1\end{pmatrix} \\ &=i-i \\ &=0 \end{aligned}

The $conjugate in$ $x^H$ is essential. Using $x^Ty$ instead would not define the standard complex inner product because $x^Tx$ need not be a nonnegative real number.

Projection in a Weighted Geometry

The projection of $x$ onto the one-dimensional subspace spanned by nonzero $u$ is

\operatorname{proj}^{A}_{u}(x)=u\frac{\langle u,x\rangle_A}{\langle u,u\rangle_A}

Its $residual is$ $A$ -orthogonal to $u$ :

\left\langle u,x-\operatorname{proj}^{A}_{u}(x)\right\rangle_A=0

This is the same geometric principle behind weighted least squares: the fitted residual is orthogonal to the model space under the chosen inner product.

Using the same $A$ as above, project $x=e_2$ onto $u=e_1$ . The weighted coefficient is $1/2$ , not the Euclidean coefficient zero:

\operatorname{proj}^{A}_{e_1}(e_2)=\frac12e_1,\qquad r=\begin{pmatrix}-1/2\\1\end{pmatrix},\qquad e_1^TAr=2(-1/2)+1=0

Writing an Inner Product as a Matrix

In the standard basis $e_1,\ldots,e_n$ , every finite-dimensional inner product has a unique representation matrix with entries

a_{ij}=\langle e_i,e_j\rangle

The matrix is symmetric positive definite over the reals and Hermitian positive definite over the complex numbers.

For vectors $v_1,\ldots,v_k$ , their Gram matrix is

G_{ij}=\langle v_i,v_j\rangle_A

The Gram matrix is Hermitian positive semidefinite. It is positive definite exactly when the vectors are $linearly independent$ . Its entries record every pairwise inner product, so they can be used to check angles and linear independence.