Least Squares Method

Choosing the Line That Best Fits the Data

Suppose a set of observations or experimental results consists of value pairs $(x, y)$ . When their scatter diagram follows an approximately straight trend, many different lines could be drawn through the point cloud.

The question is, out of the many straight lines we could draw through these points, which is the best straight line that best represents the entire dataset?

The Least Squares Method is a mathematical procedure used to find the straight line that best fits the set of data points according to a precise criterion: the smallest possible sum of squared vertical residuals. The fitted slope is unique as long as the observed $x$ values are not all identical.

Minimizing Squared Residuals

The fitted line is determined from the vertical residual at each observed point:

Prediction Line: We try drawing a straight line ( $\hat{y} = a + bx$ ) among the data points.
Residual: For each observed value $y_i$ , the line gives a predicted value . Their signed vertical difference is the :

Why square the residuals?

Squaring prevents positive and negative residuals from canceling one another.
A residual with twice the magnitude contributes four times as much to the sum, so large departures receive more weight.

Fitting Advertising and Sales Data

For example, a company wants to see the relationship between the advertising costs they incur (in millions of rupiah) and the number of products sold (in thousands of units). The data they collected is as follows:

The straight line in the diagram is the best-fit line for the advertising and sales data. The dashed lines show the residual for each point. The least-squares method selects the line with the smallest sum of squared residuals.

Finding the Line by Minimizing Squared Residuals

Mathematically, we are looking for the line with the equation:

\hat{y} = a + bx

The intercept $a$ and slope $b$ are chosen to minimize the following sum of squared residuals:

\begin{aligned} \sum e_i^2 &= \sum (y_i - \hat{y}_i)^2 \\ &= \sum (y_i - (a + bx_i))^2 \end{aligned}

Through calculus (which we don't need to derive here), formulas are found to obtain the values of $a$ and $b$ that satisfy this condition:

b = \frac{n(\sum xy) - (\sum x)(\sum y)}{n(\sum x^2) - (\sum x)^2}

Formula key:

$n$ = Number of data pairs.
$\sum x$ , $\sum y$ = Sum of all $x$ and $y$ values.
= Sum of the product of each and pair.

The least-squares method provides a systematic criterion for the line that represents the linear trend: its residual sum of squares is minimal.

If every $x$ value is the same, then $n(\sum x^2)-(\sum x)^2=0$ and the slope formula has a zero denominator. In that case, the data cannot determine a unique linear regression slope.