For AI agents: use /llms.txt for the Nakafa content index.
Imagine we are trying to fit a curve to a set of data. In many practical cases, we have more data than parameters we want to find. Such situations create what is called overdetermined linear equation systems.
This system has a special characteristic. The number of equations is greater than the number of unknown variables. Mathematically, if we have equations and variables, then the condition makes this system "overdetermined".
Let's look at a concrete example. Suppose we have data points that we want to fit with a parabola or quadratic curve.
The data we have is as follows.
We want to find a parabola with the following form.
Here we are looking for parameters, namely (quadratic coefficient), (linear coefficient), and (constant).
Now, how do we use this data to find the parabola parameters? The idea is simple. For each data point, we can write one equation. With data points, we will get equations.
Now let's calculate the square values for each . For example, for , we have . Similarly for the others. After calculating everything, our equations become like this.
The system of equations above can be written in matrix form .
In general, for a quadratic polynomial model with data points, the matrix form is as follows.
Now we face an interesting situation. In our example, matrix has size and vector has size . This means we have equations but only unknown variables.
Does this mean the system cannot be solved? Let's examine this more carefully.
The three columns of matrix are linearly independent, so the rank of matrix is . However, when we add vector to matrix to form the augmented matrix , its rank becomes .
This condition tells us something important. This system has no exact solution. In simple terms, there is no single parabola that can pass through all data points perfectly.
So what should we do? Give up? Of course not!
When an overdetermined linear equation system has no exact solution, we use the least squares approach. The basic idea makes perfect sense. If we cannot find a parabola that passes through all points, let's find the parabola that is "closest" to all points.
Mathematically, this method seeks parameters that minimize the sum of squared differences between predicted values and observed values. Imagine we draw a parabola, then measure the vertical distance from each data point to that parabola. The least squares method finds the parabola that makes the total squared distances as small as possible.
Overdetermined linear equation systems are very common in the real world, especially when we have many measurement data but a relatively simple model.
The least squares approach provides an optimal solution in the sense of minimizing overall error, making it very practical for engineering and scientific applications.
