Positive Definite Matrix

Positive and Semidefinite Definitions

Imagine we have a bowl that always faces upward. No matter from which direction we throw a ball into it, the ball will always roll to the lowest point. Positive definite matrices have a property similar to this bowl in mathematical space.

A symmetric matrix $A \in \mathbb{R}^{n \times n}$ or Hermitian matrix $A \in \mathbb{C}^{n \times n}$ is called positive semidefinite if:

A matrix is called positive definite if a stronger condition is satisfied:

Conversely, a matrix is called negative semidefinite if $-A$ is positive semidefinite, and negative definite if $-A$ is positive definite. A matrix that is neither positive nor negative semidefinite is called indefinite.

Geometric Properties of Ellipsoids

Why is this concept important? Let's look at it from an interesting geometric perspective.

If $A$ is a positive definite matrix, then the set:

forms an ellipsoid in $n$-dimensional space centered at the origin. This ellipsoid shape provides a visual representation of how the matrix "stretches" space in various directions.

Specifically, if $A = \frac{1}{r^2} I$ where $I$ is the identity matrix, then $E$ becomes a sphere with radius $r$.

Properties of Diagonal Elements

One simple but important property of positive definite matrices is that all their diagonal elements must be positive.

If $A$ is a positive definite matrix, then all diagonal elements $a_{ii} > 0$ for $i = 1, 2, \ldots, n$.

Why is this so? Because if we take the standard basis vector $e_i$ that has component 1 at position $i$ and 0 elsewhere, then:

However, this condition is not sufficient to guarantee positive definiteness. We can have a matrix with all positive diagonal elements but still not be positive definite.

Eigenvalue Criteria

The most elegant way to determine positive definiteness is through eigenvalues. Let's look at this very useful criterion.

A symmetric matrix $A \in \mathbb{R}^{n \times n}$ or Hermitian matrix $A \in \mathbb{C}^{n \times n}$ is positive definite if and only if all its eigenvalues are positive:

For positive semidefinite, all eigenvalues must be non-negative ($\lambda_i \geq 0$).

Why is this true? Because for symmetric or Hermitian matrices, we can perform orthogonal diagonalization. If $A = Q \Lambda Q^T$ where $\Lambda$ is a diagonal matrix containing eigenvalues, then:

where $y = Q^T x$. This expression is positive for all $x \neq 0$ if and only if all $\lambda_i > 0$.

Leading Principal Minor Criteria

There's another practical way to check positive definiteness without computing eigenvalues. This method is called the leading principal minor criteria or Hauptminorenkriterium.

Let $A \in \mathbb{R}^{n \times n}$ be a symmetric matrix. For $k = 1, 2, \ldots, n$, define the $k$-th leading principal minor as:

This is the upper-left submatrix of size $k \times k$ from matrix $A$. The determinant of $A_k$ is called the $k$-th leading principal minor.

A symmetric matrix $A$ is positive definite if and only if all its leading principal minors are positive:

It's important to note that this leading principal minor criterion only detects positive definiteness, not positive semidefiniteness.

Application Examples

Let's look at some examples to better understand these concepts.

Indefinite Matrix with Mixed Eigenvalues

Consider the matrix $A = \begin{pmatrix} 1 & 5 \\ 5 & 4 \end{pmatrix}$. This matrix has eigenvalues approximately $\lambda_1 \approx 7.7202$ and $\lambda_2 \approx -2.7202$.

Since there is a negative eigenvalue, this matrix is indefinite. Even though its diagonal elements (1 and 4) are both positive, this doesn't guarantee positive definiteness.

Positive Definite Matrix with Verification

Now consider the matrix $A = \begin{pmatrix} 5 & 1 \\ 1 & 4 \end{pmatrix}$. This matrix has eigenvalues $\lambda_1 \approx 5.6180$ and $\lambda_2 \approx 3.3820$. Both eigenvalues are positive, so this matrix is positive definite.

We can also verify this using the leading principal minor criteria:

Since all leading principal minors are positive, this matrix is positive definite.

Inverse of Positive Definite Matrix

From the previous example, the inverse of the positive definite matrix is:

This matrix has eigenvalues approximately $\lambda_1 \approx 0.17800$ and $\lambda_2 \approx 0.29569$. Both are positive, so $A^{-1}$ is also positive definite.

Properties of Transpose Matrices

One important result in linear algebra is the property of the matrix $A^T A$ for rectangular matrices.

If $A \in \mathbb{R}^{m \times n}$ with $m \geq n$, then the matrix $A^T A \in \mathbb{R}^{n \times n}$ is positive semidefinite. This matrix becomes positive definite if and only if $A$ has full rank (rank $n$).

Why is this so? Because for any vector $x \in \mathbb{R}^n$:

This expression equals zero only if $Ax = 0$. If $A$ has full rank, then $Ax = 0$ only for $x = 0$, so $A^T A$ is positive definite.

Spectral Transformation

A very useful concept in practice is the ability to "shift" the spectrum of a matrix.

If $A \in \mathbb{R}^{n \times n}$ is a symmetric matrix or $A \in \mathbb{C}^{n \times n}$ is a Hermitian matrix, and $t$ is a real number smaller than all eigenvalues of $A$, then the matrix:

is positive definite.

This provides a practical way to make a matrix positive definite by shifting its eigenvalues. If we know the lower bound of the smallest eigenvalue, we can shift the spectrum so that all eigenvalues become positive.

Positive definite matrices play a central role in optimization, numerical analysis, and machine learning due to their geometric properties that guarantee the existence of a unique global minimum.