Tikhonov Regularization

Why Least Squares Can Be Unstable

For $A\in\mathbb{R}^{m\times n}$ and $b\in\mathbb{R}^m$ , ordinary least squares solves

\min_x\|Ax-b\|_2^2

A minimizer always exists, even when the equations $Ax=b$ are inconsistent. If $A$ is rank deficient, that minimizer may not be unique. If $A$ is nearly rank deficient, small perturbations in $b$ can cause large changes in the fitted parameters.

The $singular value decomposition$ explains the amplification. In the minimum-norm least-squares solution, each data component is divided by its corresponding $positive singular value$ . Zero singular values contribute zero through the pseudoinverse. A very small positive singular value therefore amplifies errors in its data direction.

Tikhonov Problem

Tikhonov regularization balances $data fit$ against a chosen notion of plausible parameters:

For $\alpha>0$ , minimize:

\min_x\left(\|Ax-b\|_2^2+\alpha^2\|L(x-x_0)\|_2^2\right)

Here $x_0$ is a reference estimate, $L$ defines what should remain small or smooth, and $\alpha$ sets how strongly the solution follows the data compared with the constraint defined by $L$ .

$L=I$ penalizes distance from $x_0$ .
A first-difference matrix penalizes rapid changes between neighboring parameters.
A second-difference matrix penalizes curvature and favors smoother profiles.

The penalty adds an explicit modeling assumption about the solution and can improve the numerical conditioning of the optimization problem. The observations still contain the same information.

Augmented Least Squares Form

The regularized problem can be written as the following least-squares problem:

\min_x\left\|\begin{pmatrix}A\\\alpha L\end{pmatrix}x-\begin{pmatrix}b\\\alpha Lx_0\end{pmatrix}\right\|_2^2

Differentiating this objective with respect to $x$ gives the normal equation:

(A^TA+\alpha^2L^TL)x =A^Tb +\alpha^2L^TLx_0

This normal equation has a $unique solution$ exactly when the following condition holds:

\operatorname{null}(A)\cap\operatorname{null}(L)=\{0\}

For standard ridge regularization with $L=I$ , this condition holds for every $\alpha>0$ because $A^TA+\alpha^2I$ is positive definite.

Spectral Filtering

For $L=I$ and $x_0=0$ , write $A=U\Sigma V^T$ . Then

x_\alpha=\sum_i\frac{\sigma_i}{\sigma_i^2+\alpha^2}(u_i^Tb)v_i

When $\sigma_i$ is much larger than $\alpha$ , the factor $\phi_i(\alpha)$ is close to $1$ , so regularization changes the $corresponding solution component$ very little. When $\sigma_i$ is much smaller than $\alpha$ , the factor is close to $0$ , so that component becomes smaller. This prevents a small measurement error from causing a large change in the fitted solution.

Worked Sensitivity Example

Consider

A=\begin{pmatrix}1&0\\0&0.1\end{pmatrix},\qquad b=\begin{pmatrix}1\\0.2\end{pmatrix}

The second singular value is only $0.1$ . Changing the second observation by $0.01$ changes the unregularized second parameter by $0.1$ .

With $L=I$ , $x_0=0$ , and $\alpha=0.2$ ,

(A^TA+0.2^2I)x=A^Tb

The same $0.01$ perturbation now changes the second parameter by only $0.02$ . The estimate is $less sensitive to this perturbation$ , while the penalty shrinks the second parameter toward the reference value zero. The difference from the unregularized estimate alone does not tell us which estimate is closer to the unknown true parameter.

Compare the second parameter before and after the same data change. The first parameter is unchanged by this particular perturbation.

Method	Before	After	Change
Unregularized	$2$	$21/10$	$1/10$
$\alpha=1/5$	$2/5$	$21/50$	$1/50$

\Delta x_2=\frac{(1/10)(1/100)}{(1/10)^2+(1/5)^2}=\frac{1/1000}{1/20}=\frac1{50}

Choosing the Regularization Parameter

There is $no universally best$ $\alpha$ . It depends on the noise level, parameter scaling, penalty, and purpose of the model.

Method	Appropriate evidence	Decision
Validation or cross-validation	Predictive modeling with held-out observations	Choose the value with the best out-of-sample performance
Discrepancy principle	A credible estimate of the observation-noise norm	Match the residual norm to the expected noise level
Generalized cross-validation	Linear inverse problems without a separate validation set	Minimize a predictive score computed from the fitted system
L-curve	Exploratory inverse-problem analysis	Inspect the corner of the residual-norm versus penalty-norm curve

A small $\alpha$ prioritizes fitting the data, so noise can still be amplified. A large $\alpha$ prioritizes the penalty, which can make the fitted solution too smooth or shrink it too strongly. Scale the parameter columns and penalty rows consistently so the numerical value of $\alpha$ has a clear meaning.

General Weights and Constraints

A diagonal matrix $L=\operatorname{diag}(\omega_1,\ldots,\omega_n)$ gives parameter-specific penalties:

\|L(x-x_0)\|_2^2=\sum_{i=1}^n\omega_i^2(x_i-x_{0i})^2

Larger $\omega_i$ means stronger shrinkage toward $x_{0i}$ . Choose weights from the units and prior knowledge such as each parameter's plausible scale or an expected smooth change between neighboring parameters. Difficulty in estimating a coefficient is not enough to determine its weight.

An equality such as $x_i=c$ should be modeled as an $explicit equality constraint$ when it must hold exactly. Bounds such as $l_i\le x_i\le u_i$ require constrained least squares. An extremely large penalty can approximate an equality, but it is not a replacement for an exact constraint and can worsen numerical scaling.

Solving the Extended System Stably

Use the normal equation to derive the regularized estimator and examine how $\alpha$ and $L$ change it. For numerical computation, avoid forming $A^TA$ because that squares the condition number. Solve the $augmented least-squares system$ with $\mathrm{QR}$ or $\mathrm{SVD}$ instead. Report $\alpha$ , $L$ , parameter scaling, residual norm, and penalty norm so another reader can repeat the calculation and inspect the trade-off between data fit and penalty.