Least Squares - Quadratic Example
Thomas J. Kennedy
The previous discussion focused on linear approximation with Least Squares. However, we are not limited to lines.
1 The Data
Let us use the same three points from the previous discussion
| x | f(x) |
|---|---|
| 0 | 0 |
| 1 | 1 |
| 2 | 4 |
We want to a polynomial of degree 2 (quadratic function) that approximates this (supposedly) unknown function. We want a function in the form
\[ \begin{align} \hat{\varphi} & = \sum_{i=0}^{2} c_i x^i \\ & = c_0 + c_1 x + c_2 x^2 \\ \end{align} \]
1.1 The Set Up
The X matrix is defined by the taking each constant in our approximation function and plugging in each point. For our selected points
| x | f(x) |
|---|---|
| 0 | 0 |
| 1 | 1 |
| 2 | 4 |
and selected approximation function
\[ \hat{\varphi} = c_0 (1) + c_1 x + c_2 x^2 \]
the matrix $X$ will be defined as
$$ X = \left[\begin{array}{rrr} 1 & 0 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 4 \\ \end{array}\right] $$
- The first column is defined by taking each $x$ value and plugging it into $y=1$.
- The second column is defined by taking each $x$ value and plugging it into $y=x$. 23 The third column is defined by taking each $x$ value and plugging it into $y=x^2$.
For the $Y$ matrix, we need only copy each $y$.
$$ Y = \left[\begin{array}{r} 0 \\ 1 \\ 4 \\ \end{array}\right] $$
$X^T$ is the transpose of $X$.
$$ \left[\begin{array}{rrr} 1 & 1 & 1\\ 0 & 1 & 2\\ 0 & 1 & 4\\ \end{array}\right] $$
1.2 Constructing XTX|XTY
Once the matrices are defined, $X^{T}X$ and $X^{T}Y$ must be computed. The two matrix multiplications are left as an exercise to the reader (i.e., you).
$$ X^{T}X = \left[\begin{array}{rrr} 3 & 3 & 5\\ 3 & 5 & 9\\ 5 & 9 & 17\\ \end{array}\right] $$
$$ X^{T}Y = \left[\begin{array}{r} 5\\ 9\\ 17\\ \end{array}\right] $$
With both matrices computed, we can construct the $X^TX|X^TY$ augmented matrix. This will, in turn, allow us to compute $c_0$, $c_1$, and $c_2$.
$$ [X^{T}X|X^{T}Y] = \left[\begin{array}{rrr|r} 3 & 3 & 5 & 5\\ 3 & 5 & 9 & 9\\ 5 & 9 & 17 & 17\\ \end{array}\right] $$
1.3 Solving XTX|XTY
The system, $[X^TX|X^TY]$ can be solved using Gaussian Elimination. We will start with $[X^TX|X^TY]$.
$$ \left[\begin{array}{rrr|r} 3 & 3 & 5 & 5\\ 3 & 5 & 9 & 9\\ 5 & 9 & 17 & 17\\ \end{array}\right] $$
Our first step will be to perform two subtraction operations:
- $r_1 = r_1 - r_0$
- $r_2 = r_2 - r_0$
$$ \left[\begin{array}{rrr|r} 3 & 3 & 5 & 5\\ 0 & 2 & 4 & 4\\ 2 & 6 & 12 & 12\\ \end{array}\right] $$
The next two operations are then: $r_1 = \frac{1}{2}r_1$ and $r_2 = \frac{1}{2}r_2$
$$ \left[\begin{array}{rrr|r} 3 & 3 & 5 & 5\\ 0 & 1 & 2 & 2\\ 1 & 3 & 6 & 6\\ \end{array}\right] $$
Next, $r_0 = r_0 - r_2$
$$ \left[\begin{array}{rrr|r} 2 & 0 & -1 & -1\\ 0 & 1 & 2 & 2\\ 1 & 3 & 6 & 6\\ \end{array}\right] $$
Next, $r_2 = r_2 - 3r_1$
$$ \left[\begin{array}{rrr|r} 2 & 0 & -1 & -1\\ 0 & 1 & 2 & 2\\ 1 & 0 & 0 & 0\\ \end{array}\right] $$
That allows us to swap $r_0$ and $r_2$.
$$ \left[\begin{array}{rrr|r} 1 & 0 & 0 & 0\\ 0 & 1 & 2 & 2\\ 2 & 0 & -1 & -1\\ \end{array}\right] $$
then subtract $r_0$ from $r_2$ ($r_2 = r_2 - r_0$).
$$ \left[\begin{array}{rrr|r} 1 & 0 & 0 & 0\\ 0 & 1 & 2 & 2\\ 0 & 0 & -1 & -1\\ \end{array}\right] $$
and scale $r_2$ by $-1$.
$$ \left[\begin{array}{rrr|r} 1 & 0 & 0 & 0\\ 0 & 1 & 2 & 2\\ 0 & 0 & 1 & 1\\ \end{array}\right] $$
That leaves one last step (i.e., $r_1 = r_1 - 2r_2$).
$$ \left[\begin{array}{rrr|r} 1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 1\\ \end{array}\right] $$
1.4 Final Result
Coefficients
$$c_0 = 0$$ $$c_1 = 0$$ $$c_2 = 1$$
Approximation Function (phi hat)
\[\begin{align} \hat{\varphi} &= 0 * x^0 + 0 * x^1 + 1 * x^2\\ & = x^2 \end{align} \]