1 Another Example

Suppose we wanted to compute a discrete approximation for $f(x) = 2x^2$ where $x \ge 0$ using four points.

We will stick with a line. This leads to two basis functions:

$\pi_0 = 1$ and $\pi_1 = x$.

Using these two basis functions our approximation function takes the form:

$$ [\begin{align} \hat{\varphi} &= \sum_{i=0}^{1} c_i \pi_i \\ &= \sum_{i=0}^{1} c_i x^i \\ &= c_0 + c_1 x \ \end{align} $$

We could use the $[X^TX|X^TY]$ method… Let us try the $A\vec{c}|b$ method instead.

1.1 The Setup

Our $A$ matrix is defined as

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 \pi_{0}(x_i) \pi_{0}(x_i) & \sum\limits_{i=0}^3 \pi_{0}(x_i) \pi_{1}(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{0}(x_i) & \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{1}(x_i) \\ \end{array}\right] $$

Since $\pi_0 = 1$ this simplifies to

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 1 & \sum\limits_{i=0}^3 \pi_{1}(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) & \sum\limits_{i=0}^3 \pi_{1}(x_i) \pi_{1}(x_i) \\ \end{array}\right] $$

And… if we apply $\pi_1 = x$ we end up with

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^3 1 & \sum\limits_{i=0}^3 x_i \\ \sum\limits_{i=0}^3 x_i & \sum\limits_{i=0}^3 (x_i)^2 \\ \end{array}\right] $$

Well… $\sum\limits_{i=0}^3 1 = 4$. Hooray! That is one small sum eliminated.

$$ \left[\begin{array}{rr} 4 & \sum\limits_{i=0}^3 x_i \\ \sum\limits_{i=0}^3 x_i & \sum\limits_{i=0}^3 (x_i)^2 \\ \end{array}\right] $$

If we plug in all the $x_i$ values and compute the sums, we have

$$ \left[\begin{array}{rr} 4 & 6 \\ 6 & 14 \\ \end{array}\right] $$

That leaves $\vec{b}$, which is defined as

$$ \left[\begin{array}{r} \sum\limits_{i=0}^3 \pi_{0}(x_i) f(x_i) \\ \sum\limits_{i=0}^3 \pi_{1}(x_i) f(x_i) \\ \end{array}\right] $$

which becomes

$$ \left[\begin{array}{r} \sum\limits_{i=0}^3 f(x_i) \\ \sum\limits_{i=0}^3 x_i * f(x_i) \\ \end{array}\right] $$

Once we plug in our input points, we end up with

$$ \left[\begin{array}{r} 28 \\ 72 \\ \end{array}\right] $$

That allows us to construct an augmented matrix.

$$ \left[\begin{array}{rr|r} 4 & 6 & 28 \\ 6 & 14 & 72 \\ \end{array}\right] $$

1.2 Solving the System

Let us continue using Gaussian Elimination.

$$ \left[\begin{array}{rr|r} 4 & 6 & 28 \\ 6 & 14 & 72 \\ \end{array}\right] $$

Our first step is to scale both rows using $r_0 = \frac{1}{4}r_0$ and $r_1 = \frac{1}{6}r_1$

$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 1 & \frac{7}{3} & 12 \\ \end{array}\right] $$

Next $r_1 = r_1 - r_0$

$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 0 & \frac{5}{6} & 5 \\ \end{array}\right] $$

Our next step is to scale row 1, $r_1 = \frac{6}{5}r_1$

$$ \left[\begin{array}{rr|r} 1 & \frac{3}{2} & 7 \\ 0 & 1 & 6 \\ \end{array}\right] $$

The final step is to backsolve using $r_0 = r_0 - \frac{3}{2}r_1$

$$ \left[\begin{array}{rr|r} 1 & 0 & -2 \\ 0 & 1 & 6 \\ \end{array}\right] $$

2 Final Result

Coefficients

$$c_0 = -2$$ $$c_1 = 6$$

Approximation Function (phi hat)

$$ \begin{align} \hat{\varphi} & = -2 * x^0 + 6 * x^1 \\ & = -2 + 6x \end{align} $$

3 Error?

We have discussed a few examples of Continuous Least Squares Approximation Error. What about… Discrete Least Squares Approximation Error?

The error can be evaluated using

$$ \begin{eqnarray} E^2(x) &=& ||f(x) - \hat{\varphi}(\alpha,x)|| \\ &=& \int\limits_{\mathbb{R}} \left(f - \hat{\varphi} \right)^2 d\lambda(x) \\ &=& \int\limits_{\mathbb{R}} f^2 d\lambda(x) -2\int\limits_{\mathbb{R}} \hat{\varphi}f d\lambda(x) +\int\limits_{\mathbb{R}} \hat{\varphi}^2 d\lambda(x) \\ &=& \int\limits_{\mathbb{R}} \hat{\varphi}^2 d\lambda(x) -2\int\limits_{\mathbb{R}} \hat{\varphi}f d\lambda(x) + \int\limits_{\mathbb{R}} f^2 d\lambda(x) \\ &=& \sum\limits_{i=0}^{n-1} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n-1} \hat{\varphi}(x_i)f(x_i) + \sum\limits_{i=0}^{n-1} f^2(x_i) \\ \end{eqnarray} $$

Take note of the change to sums at the very end. In this case the Riemann-Stieltjes integrals collapse to summations, not Riemann integrals.

We know that

$$ \begin{align} \hat{\varphi} & = -2 + 6x \end{align} $$

Let us take…

and… add a few columns.

We can now evaluate the sums…

$$ \begin{eqnarray} E^2(x) &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + \sum\limits_{i=0}^{n} f^2(x_i) \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + \left(0 + 4 + 64 + 324\right) \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\sum\limits_{i=0}^{n} \hat{\varphi}(x_i)f(x_i) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\left(0 + 8 + 80 + 288\right) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) -2\left(376\right) + 392 \\ &=& \sum\limits_{i=0}^{n} \hat{\varphi}^2(x_i) + 1144 \\ &=& \left(4 + 16 + 100 + 256\right) + 1144 \\ &=& 376 + 1144 \\ &=& 1520 \\ \end{eqnarray} $$

We therefore know that…

$$ \begin{eqnarray} E(x) &=& \sqrt{1520} \\ &=& \sqrt{40 \times 38} \\ &=& \sqrt{4 \times 10 \times 2 \times 19} \\ &=& \sqrt{4 \times 5 \times 2 \times 2 \times 19} \\ &=& \sqrt{4 \times 5 \times 4 \times 19} \\ &=& \sqrt{16 \times 5 \times 19} \\ &=& 4\sqrt{5 \times 19} \\ &=& 4\sqrt{95} \\ \end{eqnarray} $$