Linear Models

Linear in the Model Parameters

A linear model relates a known input $t \in \mathbb{R}^k$ to an observed response $y \in \mathbb{R}^d$ through unknown parameters $x \in \mathbb{R}^n$ :

y=h(t)x,\qquad h(t)\in\mathbb{R}^{d\times n}

The important word is $linear in the parameters$ . For fixed $t$ , the map from $x$ to $y$ is linear. The dependence on $t$ may contain powers, logarithms, sines, or other nonlinear basis functions.

From measurements to a design matrix

Suppose we collect $M$ input-response pairs $(t_i,y_i)$ and model each observation as

y_i=h(t_i)x+\varepsilon_i,\qquad i=1,\ldots,M

The error $\varepsilon_i$ contains measurement noise and model mismatch. Stacking all responses and model blocks gives

b=\begin{pmatrix}y_1\\\vdots\\y_M\end{pmatrix}\in\mathbb{R}^{Md}

The matrix $A$ is the $design matrix$ . When the observations do not satisfy one parameter vector exactly, estimate $x$ with least squares or another residual model. The data determine the parameters uniquely when $A$ has full column rank. The condition number of $A$ separately measures the numerical sensitivity of that estimate.

Basis functions create many model shapes

Let $\phi_1(t),\ldots,\phi_n(t)$ be known basis functions. A scalar-response model can be written as

\begin{aligned} y(t) &=\sum_{j=1}^{n}x_j\phi_j(t) \\ &=\begin{pmatrix}\phi_1(t)&\cdots&\phi_n(t)\end{pmatrix}x \end{aligned}

Different choices of basis functions create different shapes while preserving linearity in $x$ .

An affine line:

\begin{aligned} y&=a+bt\\&=\begin{pmatrix}1&t\end{pmatrix}\begin{pmatrix}a\\b\end{pmatrix}\end{aligned}

A polynomial of degree $p$ :

\begin{aligned} y&=\sum_{j=0}^{p}a_jt^j\\&=\begin{pmatrix}1&t&\cdots&t^p\end{pmatrix}\begin{pmatrix}a_0\\a_1\\\vdots\\a_p\end{pmatrix}\end{aligned}

A Fourier basis:

\begin{aligned} y&=a_0+\sum_{j=1}^{p}a_j\cos(jt)\\&\quad+\sum_{j=1}^{p}b_j\sin(jt)\end{aligned}

An interaction between two inputs:

\begin{aligned} y&=a+bt+cs+dts\\&=\begin{pmatrix}1&t&s&ts\end{pmatrix}\begin{pmatrix}a\\b\\c\\d\end{pmatrix}\end{aligned}

Terms such as $t^p$ , $\sin(jt)$ , and $ts$ may be nonlinear in the inputs. They are known coefficients once the inputs are observed, so $the model remains linear in its unknown parameters$ .

Take the observations $(t,y)=(0,1),(1,3),(2,5)$ and the basis $1,t$ . Each observed input gives one row of the design matrix.

\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}\begin{pmatrix}a\\b\end{pmatrix}=\begin{pmatrix}1\\3\\5\end{pmatrix}

The third observation agrees with the first two: one parameter vector $(a,b)=(1,2)$ explains all three values.

Multiple responses

If every input produces several responses, place their parameter vectors in a matrix $B$ . For a row of basis values $\phi(t)^{\mathsf T}$ ,

y(t)^{\mathsf T}=\phi(t)^{\mathsf T}B

$Each column$ of $B$ describes one response. This is still a linear model because every output is a linear combination of the same known basis values.

The Ideal Gas Law

pV=nRT

At known temperature $T$ and volume $V$ , solving for pressure gives

p=\frac{RT}{V}n

Pressure $p$ is linear in the $unknown amount of substance$ $n$ , measured in moles. The quantity $n$ is not itself the number of molecules. The number of specified entities is $N=nN_A$ .

Repeated pressure measurements at several known $(T_i,V_i)$ values yield one design-matrix row $RT_i/V_i$ per observation.

Local linearization of a nonlinear model

A genuinely nonlinear parameter model has the form

y=g(t,x)

In this model, parameters may occur inside exponentials, powers, or products with one another. Near a current parameter value $x_0$ , a $first-order Taylor approximation$ gives

g(t,x) \approx g(t,x_0) +J_xg(t,x_0)(x-x_0)

The Jacobian $J_xg(t,x_0)$ is known after choosing $x_0$ , so the correction $x-x_0$ appears linearly. This approximation is accurate only near $x_0$ . Iterative methods such as Gauss-Newton repeatedly update the linearization point.

Checks before fitting

Before interpreting fitted parameters, verify that:

the design matrix columns represent the intended basis functions and units
the data contain enough $independent information$ for the parameters, typically $\operatorname{rank}(A)=n$
feature scales and the condition number do not make the solve numerically fragile
the chosen residual loss matches the noise and outlier assumptions
a local linearization is checked again against the original nonlinear model

A flexible basis can fit many shapes, but flexibility alone does not guarantee identifiability, numerical stability, or useful predictions.