# Nakafa Learning Content

> For AI agents: use [llms.txt](https://nakafa.com/llms.txt) for the site index. Markdown versions are available by appending `.md` to content URLs or sending `Accept: text/markdown`.

URL: https://nakafa.com/en/subjects/ai-ds/linear-methods/positive-definite-matrix
Source: https://raw.githubusercontent.com/nakafaai/nakafa.com/refs/heads/main/packages/contents/material/lesson/ai-ds/linear-methods/positive-definite-matrix/en.mdx

Learn positive definite matrices with eigenvalue criteria, leading principal minors, geometric properties, and optimization applications.

---

## Positive and Semidefinite Definitions

Imagine we have a bowl that always faces upward. No matter from which direction we throw a ball into it, the ball will always roll to the lowest point. Positive definite matrices have a property similar to this bowl in mathematical space.

A symmetric matrix $$A \in \mathbb{R}^{n \times n}$$ or Hermitian matrix $$A \in \mathbb{C}^{n \times n}$$ is called **positive semidefinite** if:

Visible text: A symmetric matrix or Hermitian matrix is called **positive semidefinite** if:

Component: MathContainer
Children:

```math
x^T A x \geq 0 \text{ for all } x \in \mathbb{R}^n
```

```math
x^H A x \geq 0 \text{ for all } x \in \mathbb{C}^n
```

A matrix is called **positive definite** if a stronger condition is satisfied:

Component: MathContainer
Children:

```math
x^T A x > 0 \text{ for all } x \neq 0 \text{ in } \mathbb{R}^n
```

```math
x^H A x > 0 \text{ for all } x \neq 0 \text{ in } \mathbb{C}^n
```

Conversely, a matrix is called **negative semidefinite** if $$-A$$ is positive semidefinite, and **negative definite** if $$-A$$ is positive definite. A matrix that is neither positive nor negative semidefinite is called **indefinite**.

Visible text: Conversely, a matrix is called **negative semidefinite** if is positive semidefinite, and **negative definite** if is positive definite. A matrix that is neither positive nor negative semidefinite is called **indefinite**.

## Geometric Properties of Ellipsoids

Why is this concept important? Let's look at it from an interesting geometric perspective.

If $$A$$ is a positive definite matrix, then the set:

Visible text: If is a positive definite matrix, then the set:

```math
E = \{x \in \mathbb{R}^n : x^T A x = 1\}
```

forms an ellipsoid in $$n$$-dimensional space centered at the origin. This ellipsoid shape provides a visual representation of how the matrix "stretches" space in various directions.

Visible text: forms an ellipsoid in -dimensional space centered at the origin. This ellipsoid shape provides a visual representation of how the matrix "stretches" space in various directions.

Specifically, if $$A = \frac{1}{r^2} I$$ where $$I$$ is the identity matrix, then $$E$$ becomes a sphere with radius $$r$$.

Visible text: Specifically, if where is the identity matrix, then becomes a sphere with radius .

## Properties of Diagonal Elements

One simple but important property of positive definite matrices is that all their diagonal elements must be positive.

If $$A$$ is a positive definite matrix, then all diagonal elements $$a_{ii} > 0$$ for $$i = 1, 2, \ldots, n$$.

Visible text: If is a positive definite matrix, then all diagonal elements for .

Why is this so? Because if we take the standard basis vector $$e_i$$ that has component $$1$$ at position $$i$$ and $$0$$ elsewhere, then:

Visible text: Why is this so? Because if we take the standard basis vector that has component at position and elsewhere, then:

```math
a_{ii} = e_i^T A e_i > 0
```

However, this condition is not sufficient to guarantee positive definiteness. We can have a matrix with all positive diagonal elements but still not be positive definite.

## Eigenvalue Criteria

The most elegant way to determine positive definiteness is through eigenvalues. Let's look at this very useful criterion.

A symmetric matrix $$A \in \mathbb{R}^{n \times n}$$ or Hermitian matrix $$A \in \mathbb{C}^{n \times n}$$ is positive definite if and only if all its eigenvalues are positive:

Visible text: A symmetric matrix or Hermitian matrix is positive definite if and only if all its eigenvalues are positive:

```math
\lambda_1, \lambda_2, \ldots, \lambda_n > 0
```

For positive semidefinite, all eigenvalues must be non-negative ($$\lambda_i \geq 0$$).

Visible text: For positive semidefinite, all eigenvalues must be non-negative ().

Why is this true? Because for symmetric or Hermitian matrices, we can perform orthogonal diagonalization. If $$A = Q \Lambda Q^T$$ where $$\Lambda$$ is a diagonal matrix containing eigenvalues, then:

Visible text: Why is this true? Because for symmetric or Hermitian matrices, we can perform orthogonal diagonalization. If where is a diagonal matrix containing eigenvalues, then:

```math
x^T A x = x^T Q \Lambda Q^T x = y^T \Lambda y = \sum_{i=1}^n \lambda_i y_i^2
```

where $$y = Q^T x$$. This expression is positive for all $$x \neq 0$$ if and only if all $$\lambda_i > 0$$.

Visible text: where . This expression is positive for all if and only if all .

## Leading Principal Minor Criteria

There's another practical way to check positive definiteness without computing eigenvalues. This method is called the **leading principal minor criteria** or Hauptminorenkriterium.

Let $$A \in \mathbb{R}^{n \times n}$$ be a symmetric matrix. For $$k = 1, 2, \ldots, n$$, define the $$k$$-th leading principal minor as:

Visible text: Let be a symmetric matrix. For , define the -th leading principal minor as:

```math
A_k = \begin{pmatrix} a_{11} & \cdots & a_{1k} \\ \vdots & \ddots & \vdots \\ a_{k1} & \cdots & a_{kk} \end{pmatrix}
```

This is the upper-left submatrix of size $$k \times k$$ from matrix $$A$$. The determinant of $$A_k$$ is called the **$$k$$-th leading principal minor**.

Visible text: This is the upper-left submatrix of size from matrix . The determinant of is called the **-th leading principal minor**.

A symmetric matrix $$A$$ is positive definite if and only if all its leading principal minors are positive:

Visible text: A symmetric matrix is positive definite if and only if all its leading principal minors are positive:

```math
\det A_k > 0 \text{ for all } k = 1, 2, \ldots, n
```

> It's important to note that this leading principal minor criterion only detects positive definiteness, not positive semidefiniteness.

## Application Examples

Let's look at some examples to better understand these concepts.

### Indefinite Matrix with Mixed Eigenvalues

Consider the matrix $$A = \begin{pmatrix} 1 & 5 \\ 5 & 4 \end{pmatrix}$$. This matrix has eigenvalues approximately $$\lambda_1 \approx 7.7202$$ and $$\lambda_2 \approx -2.7202$$.

Visible text: Consider the matrix . This matrix has eigenvalues approximately and .

Since there is a negative eigenvalue, this matrix is **indefinite**. Even though its diagonal elements ($$1$$ and $$4$$) are both positive, this doesn't guarantee positive definiteness.

Visible text: Since there is a negative eigenvalue, this matrix is **indefinite**. Even though its diagonal elements ( and ) are both positive, this doesn't guarantee positive definiteness.

### Positive Definite Matrix with Verification

Now consider the matrix $$A = \begin{pmatrix} 5 & 1 \\ 1 & 4 \end{pmatrix}$$. This matrix has eigenvalues $$\lambda_1 \approx 5.6180$$ and $$\lambda_2 \approx 3.3820$$. Both eigenvalues are positive, so this matrix is **positive definite**.

Visible text: Now consider the matrix . This matrix has eigenvalues and . Both eigenvalues are positive, so this matrix is **positive definite**.

We can also verify this using the leading principal minor criteria:

Component: MathContainer
Children:

```math
A_1 = (5), \quad \det A_1 = 5 > 0
```

```math
A_2 = \begin{pmatrix} 5 & 1 \\ 1 & 4 \end{pmatrix}, \quad \det A_2 = 20 - 1 = 19 > 0
```

Since all leading principal minors are positive, this matrix is positive definite.

### Inverse of Positive Definite Matrix

From the previous example, the inverse of the positive definite matrix is:

```math
A^{-1} = \begin{pmatrix} 4/19 & -1/19 \\ -1/19 & 5/19 \end{pmatrix}
```

This matrix has eigenvalues approximately $$\lambda_1 \approx 0.17800$$ and $$\lambda_2 \approx 0.29569$$. Both are positive, so $$A^{-1}$$ is also positive definite.

Visible text: This matrix has eigenvalues approximately and . Both are positive, so is also positive definite.

## Properties of Transpose Matrices

One important result in linear algebra is the property of the matrix $$A^T A$$ for rectangular matrices.

Visible text: One important result in linear algebra is the property of the matrix for rectangular matrices.

If $$A \in \mathbb{R}^{m \times n}$$ with $$m \geq n$$, then the matrix $$A^T A \in \mathbb{R}^{n \times n}$$ is **positive semidefinite**. This matrix becomes **positive definite** if and only if $$A$$ has full rank (rank $$n$$).

Visible text: If with , then the matrix is **positive semidefinite**. This matrix becomes **positive definite** if and only if has full rank (rank ).

Why is this so? Because for any vector $$x \in \mathbb{R}^n$$:

Visible text: Why is this so? Because for any vector :

```math
x^T (A^T A) x = (Ax)^T (Ax) = \|Ax\|^2 \geq 0
```

This expression equals zero only if $$Ax = 0$$. If $$A$$ has full rank, then $$Ax = 0$$ only for $$x = 0$$, so $$A^T A$$ is positive definite.

Visible text: This expression equals zero only if . If has full rank, then only for , so is positive definite.

## Spectral Transformation

A very useful concept in practice is the ability to "shift" the spectrum of a matrix.

If $$A \in \mathbb{R}^{n \times n}$$ is a symmetric matrix or $$A \in \mathbb{C}^{n \times n}$$ is a Hermitian matrix, and $$t$$ is a real number smaller than all eigenvalues of $$A$$, then the matrix:

Visible text: If is a symmetric matrix or is a Hermitian matrix, and is a real number smaller than all eigenvalues of , then the matrix:

```math
A - tI
```

is positive definite.

This provides a practical way to make a matrix positive definite by shifting its eigenvalues. If we know the lower bound of the smallest eigenvalue, we can shift the spectrum so that all eigenvalues become positive.

> Positive definite matrices play a central role in optimization, numerical analysis, and machine learning due to their geometric properties that guarantee the existence of a unique global minimum.