Identifiability and Matrix Rank

Identifiability in a Linear Model

A parameter is identifiable when the available data $determine it uniquely$ . Let $A \in \mathbb{R}^{m \times n}$ , $b \in \mathbb{R}^m$ , and $m \geq n$ . The linear least-squares problem is

\min_{x \in \mathbb{R}^n} \|Ax-b\|_2^2

The corresponding normal equations are

A^T A x = A^T b

The parameter vector $x$ is uniquely identifiable exactly when $A$ has full column rank:

\operatorname{rank}(A)=n

These statements are equivalent. If $z \in \ker(A)$ is nonzero, then

A(x+z)=Ax

This equality shows that the data cannot distinguish $x$ from $x+z$ . If $\ker(A)=\{0\}$ , then $A^TA$ is positive definite and the least-squares problem has one minimizer.

For a concrete example, the two columns of the following matrix are identical. Only the sum of the parameters affects the prediction.

A=\begin{pmatrix}1&1\\2&2\end{pmatrix},\qquad Ax=\begin{pmatrix}x_1+x_2\\2(x_1+x_2)\end{pmatrix}

Parameter vector	Prediction
$(1,0)^T$	$(1,2)^T$
$(0,1)^T$	$(1,2)^T$
$(1/2,1/2)^T$	$(1,2)^T$

A\begin{pmatrix}1\\-1\end{pmatrix}=\begin{pmatrix}0\\0\end{pmatrix}

Solution Uniqueness and Numerical Stability

$Full column rank guarantees a unique least-squares minimizer$ , but not a numerically reliable one. A very small nonzero singular value can make the estimate highly sensitive to small changes in the data.

Question	Diagnostic	Meaning
Is the parameter structurally identifiable?	$\operatorname{rank}(A)=n$	No two parameter vectors produce the same prediction.
Is the estimate numerically well determined?	$\sigma_n$ is not tiny relative to $\sigma_1$	Small perturbations do not cause large parameter changes.
How sensitive is the problem?	$\kappa_2(A)=\sigma_1/\sigma_n$	A large condition number signals strong sensitivity.

Structural identifiability is an exact algebraic property. Practical identifiability also depends on scale, noise, and the numerical tolerance used.

Factorizations for Determining Rank

Two common $factorizations can determine rank$ without relying on determinant calculations that are sensitive to numerical error:

Method	Strength	Typical use
$\mathrm{QR}$ factorization with column pivoting	Usually cheaper than a full $\mathrm{SVD}$	Fast least-squares solving and practical rank detection
Singular value decomposition	Separates the independent directions and the scale of each direction	Detects small singular values that indicate a matrix is nearly rank deficient

Plain $\mathrm{LU}$ factorization is primarily a square-system tool. For a rectangular design matrix, rank is usually determined with pivoted $\mathrm{QR}$ or $\mathrm{SVD}$ .

Singular Value Decomposition

Let $q=\min(m,n)$ . Every matrix $A \in \mathbb{R}^{m \times n}$ has a singular value decomposition

A=U\Sigma V^T

In this decomposition, $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal. The matrix $\Sigma \in \mathbb{R}^{m \times n}$ has nonnegative diagonal entries satisfying

\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_q \geq 0

The exact rank is the $number of positive singular values$ :

\operatorname{rank}(A)=\#\{i:\sigma_i>0\}

For $m \geq n$ , full column rank is therefore equivalent to $\sigma_n>0$ . Parameter directions that the data cannot identify are spanned by the right singular vectors associated with zero singular values, which form a basis for $\ker(A)$ .

Exact Rank and Numerical Rank

If the exact rank is $r$ with $0<r<q$ , then

\sigma_r>0=\sigma_{r+1}=\cdots=\sigma_q

Floating-point data rarely contain exact zeros. A numerical rank instead counts singular values above a $scale-aware threshold$ $\tau$ :

\tau=c\,\max(m,n)\,\varepsilon_{\mathrm{mach}}\,\sigma_1

The factor $c$ reflects the application and noise model, so there is no universal decimal cutoff. If $A$ is rank deficient, the least-squares minimum still exists, but the minimizing parameter vector is not unique. The Moore-Penrose pseudoinverse selects the minimum-norm solution:

x^{\dagger}=V\Sigma^{\dagger}U^T b

Estimating Matrix Rank with SVD or Rank Revealing QR

The identity

A^TA=V(\Sigma^T\Sigma)V^T

This identity shows why squared singular values are eigenvalues of $A^TA$ . For a full-column-rank matrix in numerical computation, explicitly forming $A^TA$ $squares the condition number$ :

\kappa_2(A^TA)=\kappa_2(A)^2

Numerical software therefore usually uses $\mathrm{SVD}$ or rank-revealing $\mathrm{QR}$ to estimate matrix rank. Both algorithms work directly on $A$ , so they do not square the condition number as forming $A^TA$ does.