Linear Regression Concept

Summarizing a Straight Trend in Paired Data

A scatter diagram shows whether paired values of two variables, $X$ and $Y$, form a common pattern.

When the point cloud follows an approximately increasing or decreasing straight-line trend, we can summarize that trend with a fitted line. This line is the linear regression line, and the procedure used to determine it is linear regression.

The Best Fit Line

The linear regression line is often called the best-fit line. Under the least squares criterion, it is selected because it has the smallest possible sum of squared vertical residuals. Data points may lie above or below the line. The regression line summarizes the linear trend in the data.

Example of a Regression Line

Compare the study time and exam score in each pair below. As study time increases, the points tend to rise. That pattern gives the point cloud a positive linear trend.

The plotted regression line shows the general trend: as study time increases, the predicted exam score tends to increase. Individual scores can still lie above or below the line.

Predictions from the Regression Line

One of its main uses is prediction. For example, we could use the line to estimate the exam score of a student who studies for $7 \text{ hours}$. However, the observed study times only run from $1$ to $6$ hours. A prediction at $7 \text{ hours}$ is therefore an extrapolation and is less reliable than a prediction inside the observed range.

Least Squares Method

The linear regression line (the best-fit line) is found using a method called the Least Squares Method. The idea is to find the straight line that minimizes the sum of the squared vertical distances from each data point to the line.

Mathematically, the linear regression line has the form:

Where:

• $\hat{y}$ (read: $y$-hat) is the predicted value of $y$ by the regression line.
• $x$ is the value of the independent variable.
• is the of the line, indicating how much changes for each one-unit change in .

The values of $b$ and $a$ are calculated from the $(x, y)$ data we have using the following formulas:

Formula key:

• $n$ is the number of data pairs.
• $\sum x$ is the sum of all $x$ values.
• $\sum y$ is the sum of all $y$ values.

These formulas determine the unique least-squares line when the observed $x$ values are not all equal. If every $x$ value is identical, the slope denominator is zero and the data do not determine a unique regression slope.