Least Squares Closing Thoughts - Discrete
Thomas J. Kennedy
We now have a few more tools available to us. It is now possible to skip the initial $X^T * X$ and $X^T * Y$ matrix multiplications.
1 Another Example
Suppose we wanted to compute a discrete approximation for $f(x) = 2x^2$ where $x \ge 0$ using four points.
x | f(x) |
---|---|
0 | 0 |
1 | 2 |
2 | 8 |
3 | 18 |
We will stick with a line. This leads to two basis functions:
$\pi_0 = 1$ and $\pi_1 = x$.
Using these two basis functions our approximation function takes the form:
$$ [\begin{align} \hat{\varphi} &= \sum_{i=0}^{1} c_i \pi_i \\ &= \sum_{i=0}^{1} c_i x^i \\ &= c_0 + c_1 x \ \end{align} $$
We could use the $[X^TX|X^TY]$ method… Let us try the $A\vec{c}|b$ method instead.
1.1 The Setup
Our $A$ matrix is defined as
$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 \pi_{0}(x_i) \pi_{0}(x_i) & \sum\limits_{i=0}^3 \pi_{0}(x_i) \pi_{1}(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{0}(x_i) & \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{1}(x_i) \\ \end{array}\right] $$
Since $\pi_0 = 1$ this simplifies to
$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 1 & \sum\limits_{i=0}^3 \pi_{1}(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) & \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{1}(x_i) \\ \end{array}\right] $$
And… if we apply $\pi_1 = x$ we end up with
$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 1 & \sum\limits_{i=0}^3 x_i \\ \sum\limits_{i=0}^3 x_i & \sum\limits_{i=0}^3 (x_i)^2 \\ \end{array}\right] $$
Well… $\sum\limits_{i=0}^3 1 = 4$. Hooray! That is one small sum eliminated.
$$ \left[\begin{array}{rr} 4 & \sum\limits_{i=0}^3 x_i \\ \sum\limits_{i=0}^3 x_i & \sum\limits_{i=0}^3 (x_i)^2 \\ \end{array}\right] $$
If we plug in all the $x_i$ values and compute the sums, we have
$$ \left[\begin{array}{rr} 4 & 6 \\ 6 & 14 \\ \end{array}\right] $$
That leaves $\vec{b}$, which is defined as
$$ \left[\begin{array}{r} \sum\limits_{i=0}^3 \pi_{0}(x_i) f(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) f(x_i) \\ \end{array}\right] $$
which becomes
$$ \left[\begin{array}{r} \sum\limits_{i=0}^3 f(x_i) \\ \sum\limits_{i=0}^3 x_i * f(x_i) \\ \end{array}\right] $$
Once we plug in our input points, we end up with
$$ \left[\begin{array}{r} 28 \\ 72 \\ \end{array}\right] $$
That allows us to construct an augmented matrix.
$$ \left[\begin{array}{rr|r} 4 & 6 & 28 \\ 6 & 14 & 72 \\ \end{array}\right] $$
1.2 Solving the System
Let us continue using Gaussian Elimination.
$$ \left[\begin{array}{rr|r} 4 & 6 & 28 \\ 6 & 14 & 72 \\ \end{array}\right] $$
Our first step is to scale both rows using $r_0 = \frac{1}{4}r_0$ and $r_1 = \frac{1}{6}r_1$
$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 1 & \frac{7}{3} & 12 \\ \end{array}\right] $$
Next $r_1 = r_1 - r_0$
$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 0 & \frac{5}{6} & 5 \\ \end{array}\right] $$
Our next step is to scale row 1, $r_1 = \frac{6}{5}r_1$
$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 0 & 1 & 6 \\ \end{array}\right] $$
The final step is to backsolve using $r_0 = r_0 - \frac{3}{2}r_1$
$$ \left[\begin{array}{rr|r} 1 & 0 & -2 \\ 0 & 1 & 6 \\ \end{array}\right] $$
2 Final Result
Coefficients
$$c_0 = -2$$ $$c_1 = 6$$
Approximation Function (phi hat)
$$ \begin{align} \hat{\varphi} & = -2 * x^0 + 6 * x^1 \\ & = -2 + 6x \end{align} $$
3 Error?
We have discussed a few examples of Continuous Least Squares Approximation Error. What about… Discrete Least Squares Approximation Error?
The error can be evaluated using
$$ \begin{eqnarray} E^2(x) &=& ||f(x) - \hat{\varphi}(\alpha,x)|| \\ &=& \int\limits_{\mathbb{R}} \left(f - \hat{\varphi} \right)^2 d\lambda(x) \\ &=& \int\limits_{\mathbb{R}} f^2 d\lambda(x) -2\int\limits_{\mathbb{R}} \hat{\varphi}f d\lambda(x) +\int\limits_{\mathbb{R}} \hat{\varphi}^2 d\lambda(x) \\ &=& \int\limits_{\mathbb{R}} \hat{\varphi}^2 d\lambda(x) -2\int\limits_{\mathbb{R}} \hat{\varphi}f d\lambda(x) + \int\limits_{\mathbb{R}} f^2 d\lambda(x) \\ &=& \sum\limits_{i=0}^{n-1} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n-1} \hat{\varphi}(x_i)f(x_i) + \sum\limits_{i=0}^{n-1} f^2(x_i) \\ \end{eqnarray} $$
Take note of the change to sums at the very end. In this case the Riemann-Stieltjes integrals collapse to summations, not Riemann integrals.
We know that
$$ \begin{align} \hat{\varphi} & = -2 + 6x \end{align} $$
Let us take…
x | f(x) |
---|---|
0 | 0 |
1 | 2 |
2 | 8 |
3 | 18 |
and… add a few columns.
$x$ | $f(x)$ | $\hat{\varphi}(x)$ | $\hat{\varphi}(x)f(x)$ | $\hat{\varphi}^2(x)$ | $f^2(x)$ |
---|---|---|---|---|---|
0 | 0 | -2 | 0 | 4 | 0 |
1 | 2 | 4 | 8 | 16 | 4 |
2 | 8 | 10 | 80 | 100 | 64 |
3 | 18 | 16 | 288 | 256 | 324 |
We can now evaluate the sums…
$$ \begin{eqnarray} E^2(x) &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + \sum\limits_{i=0}^{n} f^2(x_i) \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + \left(0 + 4 + 64 + 324\right) \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\left(0 + 8 + 80 + 288\right) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\left(376\right) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) + 1144 \\ &=& \left(4 + 16 + 100 + 256\right) + 1144 \\ &=& 376 + 1144 \\ &=& 1520 \\ \end{eqnarray} $$
We therefore know that…
$$ \begin{eqnarray} E(x) &=& \sqrt{1520} \\ &=& \sqrt{40 \times 38} \\ &=& \sqrt{4 \times 10 \times 2 \times 19} \\ &=& \sqrt{4 \times 5 \times 2 \times 2 \times 19} \\ &=& \sqrt{4 \times 5 \times 4 \times 19} \\ &=& \sqrt{16 \times 5 \times 19} \\ &=& 4\sqrt{5 \times 19} \\ &=& 4\sqrt{95} \\ \end{eqnarray} $$