Principal Component Analysis

Let $X\in\mathbb{R}^{m\times n}$ contain $m$ observations of $n$ features. $\mathrm{PCA}$ begins by subtracting each feature mean:

It is symmetric positive semidefinite. It may be singular when features are linearly dependent or when $n>m-1$ .

Centering is mandatory. Standardizing each feature to unit variance is a separate modeling choice that is often appropriate when features use incomparable units. It changes the question from directions of largest absolute variance to directions of largest standardized variance.

Each later direction maximizes the same variance subject to being orthogonal to all earlier directions. The transformed coordinates, or scores, are

The sample covariance of the scores $Z$ is diagonal:

The principal components are therefore uncorrelated. They are not generally independent. Independence follows in special models, including a jointly Gaussian distribution.

The proportion of total sample variance explained by component $j$ is

Keeping the first $k$ eigenvectors in $V_k$ gives the scores and reconstruction

Among all rank- $k$ linear reconstructions, this choice minimizes squared reconstruction error. With the sample-covariance convention above,

$\mathrm{PCA}$ can also be computed through the following thin singular value decomposition:

The right singular vectors give the principal directions, while the eigenvalues satisfy:

Using $\mathrm{SVD}$ avoids forming $X_c^TX_c$ , which would square the condition number. $\mathrm{SVD}$ also works well when $X_c$ is rectangular or rank deficient.

The covariance ellipse is rotated by the following angle:

The variances along the two principal axes are $9$ and $2.25$ . The rotation and covariance matrices are:

The variance ratio explained by the first component is calculated from the two eigenvalues:

The result $80\%$ is the fraction of the total variance explained by the first component.

Check	Reason
Center with training-set means only	prevents data leakage and preserves the learned coordinate system
Review feature units and scaling	high numerical variance can otherwise dominate for the wrong reason
Choose $k$ from validation needs	explained variance does not measure predictive relevance
Remember eigenvector signs are arbitrary	$v_j$ and $-v_j$ define the same component
Inspect nonlinear structure and outliers	$\mathrm{PCA}$ is a linear, variance-based summary and can be sensitive to both

$\mathrm{PCA}$ identifies directions with large variance. Variance alone does not establish a cause. One principal component may combine many original features. A direction with low variance may still carry a signal needed by a later prediction or classification task.

Covariance ellipse and principal directions

v_1,v_2

The cyan ellipse has semiaxis lengths

\sqrt{\lambda_1}=3

and

\sqrt{\lambda_2}=1.5

. The orange directions are its eigenvectors.

Principal Component Analysis

Start with Centered Data

Principal Component Analysis

Start with Centered Data

Principal Directions

Explained Variance and Dimension Reduction

Equivalent SVD Computation

Worked Two Dimensional Covariance

Interpretation Boundaries