1 What About The Continuous Case?

What about the continuous case?

• I am no longer limitted to picking points.
• I can use $ x \in [0, \alpha]$ where $\alpha > 0$.

Let us try to approximate $f(x) = 2x^2$ once more. This time for the entire domain $[0, \alpha]$. We will use a different set of basis functions this time…

$\pi_0 = 1$ and $\pi_1 = x$.

Using these two basis functions our approximation function takes the form:

\[\begin{align} \hat{\varphi} &= c_0 + c_1 x \ \end{align} \]

We have shifted from a discrete set of points (discrete) to an entire domain (continuous). For the continuous case, we will be working with integrals.

$$ \left[\begin{array}{rr|r} \int\limits_{0}^{\alpha} \pi_{0} \pi_{0} dx & \int\limits_{0}^{\alpha} \pi_{0} \pi_{1}dx & \int\limits_{0}^{\alpha} \pi_{0}fdx \\ \int\limits_{0}^{\alpha} \pi_{1} \pi_{0} dx & \int\limits_{0}^{\alpha} \pi_{1} \pi_{1}dx & \int\limits_{0}^{\alpha} \pi_{1}fdx \\ \end{array}\right] $$

Notice how I jumped straight to the augmented $[A|\vec{b}]$ matrix? The setup mirrors the discrete process. Based on a quick count, we have six integrals to evaluate.

1.1 Evaluating Some Integrals (i.e., Having Some Fun)

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{0}^{2}dx &=& \int\limits_{0}^{\alpha} 1dx &=& \left.x\right\vert_{x=0}^{\alpha} &=& \alpha \\ \end{align} $$

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{0}\pi_{1}dx &=& \int\limits_{0}^{\alpha} xdx &=& \left. \frac{1}{2}x^2 \right\vert_{x=0}^{\alpha} &=& \frac{1}{2}\alpha^2\\ \end{align} $$

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{1}\pi_{0}dx &=& \int\limits_{0}^{\alpha}\pi_{0}\pi_{1}dx &=& \frac{1}{2}\alpha^2\\ \end{align} $$

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{1}\pi_{1}dx &=& \int\limits_{0}^{\alpha} x^2dx &=& \left. \frac{1}{3}x^3 \right\vert_{x=0}^{\alpha} &=& \frac{1}{3}\alpha^3\\ \end{align} $$

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{0}fdx &=& \int\limits_{0}^{\alpha} 2x^2dx &=& \left. \frac{2}{3}x^3 \right\vert_{x=0}^{\alpha} &=& \frac{2}{3}\alpha^3\\ \end{align} $$

$$ \begin{align} \int\limits_{0}^{\alpha}\pi_{1}fdx &=& \int\limits_{0}^{\alpha} 2x^3dx &=& \left. \frac{1}{2}x^4 \right\vert_{x=0}^{\alpha} &=& \frac{1}{2}\alpha^4\\ \end{align} $$

1.2 Updating & Solving A|b

Now that we have evaluated all six integrals, we can take

$$ \left[\begin{array}{rr|r} \int\limits_{0}^{\alpha} \pi_{0} \pi_{0} dx & \int\limits_{0}^{\alpha} \pi_{0} \pi_{1}dx & \int\limits_{0}^{\alpha} \pi_{0}fdx \\ \int\limits_{0}^{\alpha} \pi_{1} \pi_{0} dx & \int\limits_{0}^{\alpha} \pi_{1} \pi_{1}dx & \int\limits_{0}^{\alpha} \pi_{1}fdx \\ \end{array}\right] $$

and plug in in our results.

$$ \left[\begin{array}{rr|r} \alpha & \frac{1}{2}\alpha^{2} & \frac{2}{3}\alpha^{3} \\ \frac{1}{2}\alpha^{2} & \frac{1}{3}\alpha^{3} & \frac{1}{2}\alpha^{4} \\ \end{array}\right] $$

And… now we solve the system. Let us start by scaling both rows using:

• $r_0 = \alpha^{-1}r_0$
• $r_1 = 2\alpha^{-2}r_1$

$$ \left[\begin{array}{rr|r} 1 & \frac{1}{2}\alpha & \frac{2}{3}\alpha^{2} \\ 1 & \frac{2}{3}\alpha & \alpha^{2} \\ \end{array}\right] $$

The second step is elimination ($r_1 = r_1 - r_0$).

$$ \left[\begin{array}{rr|r} 1 & \frac{1}{2}\alpha & \frac{2}{3}\alpha^{2} \\ 0 & \frac{1}{6}\alpha & \frac{1}{3}\alpha^{2} \\ \end{array}\right] $$

And… let us scale row 1 using $r_1 = \frac{6}{\alpha}$.

$$ \left[\begin{array}{rr|r} 1 & \frac{1}{2}\alpha & \frac{2}{3}\alpha^{2} \\ 0 & 1 & 2\alpha \\ \end{array}\right] $$

The last step is to backsolve. Since this is a 2x2 matrix (i.e., with 2 unknowns) we need a single elimination, $r_0 = r_0 - \frac{1}{2}\alpha r_1$.

$$ \left[\begin{array}{rr|r} 1 & 0 & \frac{-1}{3}\alpha^{2} \\ 0 & 1 & 2\alpha \\ \end{array}\right] $$

1.3 Final Result

Coefficients

$$c_0 = -\frac{1}{3}\alpha^2$$ $$c_1 = 2\alpha$$

Approximation Function (phi hat)

\[\begin{align} \hat{\varphi}(\alpha,x) & = -\frac{1}{3}\alpha^2 + 2\alpha x \\ \end{align} \]

Notice how $\hat{\varphi}$ is now a function of $\alpha$ and $x$? What we have is a function that can be used to approximate the domain $[0, \alpha]$ for and $\alpha$. Since $\alpha \in \mathbb{R}_{\geq 0}$ that is an infinite number of functions.

2 Evaluating the Error

How do we evaluate the error when using $ \hat{\varphi}(\alpha,x) = -\frac{1}{3}\alpha^2 + 2\alpha x $ to approximate $ f(x) = 2x^2 $ for $x \geq 0$?

We need to use $||f - \hat{\varphi}(\alpha,x)||^2$. Look familiar? Hopefully it does. This is the error metric we defined for least squares. It is also what we used to derive the normal equations and define $[A\vec{c}|\vec{b}]$.

We will use the same tricks as before (e.g., properties of inner products).

\[\begin{align} ||f - \hat{\varphi}(\alpha,x)||^2 & = (f-\hat{\varphi}, f-\hat{\varphi}) \\ & = (\hat{\varphi}, \hat{\varphi}) - 2(\hat{\varphi}, f) + (f, f) \\ & = \int_{\mathbb{R}} \hat{\varphi}^2 d\lambda - 2\int_{\mathbb{R}} \hat{\varphi}fd\lambda + \int_{\mathbb{R}} f^2 d\lambda \\ & = \int\limits_{x=0}^{\alpha} \hat{\varphi}^2 dx - 2\int\limits_{x=0}^{\alpha} \hat{\varphi}fdx + \int\limits_{x=0}^{\alpha} f^2 dx \\ \end{align} \]

That leaves us with three Riemann integrals. And no… I am not going to leave the three integrals as an exercise to the reader. Let us take a moment and collect the pieces we will need.

$$ \hat{\varphi}(\alpha,x) = -\frac{1}{3}\alpha^2 + 2\alpha x $$ and $$ f(x) = 2x^2 $$

Let us plug in $\hat{\varphi}$ and $f$, then continue…

\[\begin{align} ||f - \hat{\varphi}(\alpha,x)||^2 &= \int\limits_{x=0}^{\alpha} \hat{\varphi}^2 dx - 2\int\limits_{x=0}^{\alpha} \hat{\varphi}fdx + \int\limits_{x=0}^{\alpha} f^2 dx \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\int\limits_{x=0}^{\alpha} 2x^{2}(-\frac{1}{3}\alpha^2 + 2\alpha x)dx + \int\limits_{x=0}^{\alpha} (2x^2)^2 dx \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\int\limits_{x=0}^{\alpha} (-\frac{2}{3}\alpha^2x^2 + 4\alpha x^3)dx + \int\limits_{x=0}^{\alpha} 4x^4 dx \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx +\int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) + \int\limits_{x=0}^{\alpha} 4x^4 dx \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx +\int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) + \left. \frac{4}{5}x^5 \right\vert_{x=0}^{\alpha} \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx +\int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) + \frac{4}{5}(\alpha^{5} - 0^{5}) \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx +\int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) + \frac{4}{5}\alpha^{5} \\ \end{align} \]

Notice how I evaluated the rightmost integral immediately? I see no reason to delay a quick integral ($4x^4$ can be integrated with the power rule). However, the remaining three integrals require a little more legwork. These are also where most mistakes will be made.

Let us tackle the middle two integrals simultaneously.

\[ \begin{align} -2\int\limits_{x=0}^{\alpha} \hat{\varphi}fdx &= -2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx + \int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) \\ & = -2\Big(-2\alpha^{2}\int\limits_{x=0}^{\alpha} \frac{1}{3}x^{2} dx + 4\alpha \int\limits_{x=0}^{\alpha} x^3dx \Big) \\ & = -2\Big(-2\alpha^{2}\big(\frac{1}{9}x^{3}\big\vert_{x=0}^{\alpha} \big) + 4\alpha\big(\frac{1}{4} x^4 \big\vert_{x=0}^{\alpha}\big) \Big) \\ & = -2\Big(-\frac{2}{9}\alpha^{2}x^{3}\big\vert_{x=0}^{\alpha} + \alpha x^4\big\vert_{x=0}^{\alpha} \Big) \\ & = -2\Big(-\frac{2}{9}\alpha^{2}\alpha^{3} + \alpha \alpha^4\Big) \\ & = -2\Big(-\frac{2}{9}\alpha^{5} + \alpha^{5}\Big) \\ & = -2\Big(\frac{7}{9}\alpha^{5}\Big) \\ & = -\frac{14}{9}\alpha^{5} \\ \end{align} \]

We can now update our larger overall equation.

\[\begin{align} E^{2}(\alpha,x) &= ||f - \hat{\varphi}(\alpha,x)||^2 \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - 2\Big(\int\limits_{x=0}^{\alpha} -\frac{2}{3}\alpha^{2}x^{2} dx +\int\limits_{x=0}^{\alpha} 4\alpha x^3dx \Big) + \frac{4}{5}\alpha^{5} \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - \frac{14}{9}\alpha^{5} + \frac{4}{5}\alpha^{5} \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - \frac{70}{45}\alpha^{5} + \frac{36}{45}\alpha^{5} \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - \frac{34}{45}\alpha^{5} \end{align} \]

We are now left with one integral. We could expand the $(…)^2$ then integrate. However, let us review u-substitution. We will start by defining u.

$$ u = (-\frac{1}{3}\alpha^2 + 2\alpha x) $$

That leads to

\[ \begin{align} du & = 2\alpha dx \\ \frac{1}{2\alpha} du & = dx \\ \end{align} \]

It is time to finish the last integral!

\[ \begin{align} \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx &= \frac{1}{2\alpha}\int\limits_{x=0}^{\alpha} u^2 du \\ &= \frac{1}{2\alpha}\frac{1}{3}u^{3} \vert_{x=0}^{\alpha} \\ &= \frac{1}{6\alpha}u^{3} \vert_{x=0}^{\alpha} \\ &= \frac{1}{6\alpha}(-\frac{1}{3}\alpha^2 + 2\alpha x)^{3} \vert_{x=0}^{\alpha} \\ &= \frac{1}{6\alpha}\Big((-\frac{1}{3}\alpha^2 + 2\alpha (\alpha))^{3} -(-\frac{1}{3}\alpha^2 + 2\alpha (0))^{3}\Big) \\ &= \frac{1}{6\alpha}\Big((-\frac{1}{3}\alpha^2 + 2\alpha^{2})^{3} -(-\frac{1}{3}\alpha^2)^{3}\Big) \\ &= \frac{1}{6\alpha}\Big((-\frac{1}{3}\alpha^2 + \frac{6}{3}\alpha^{2})^{3} -(-\frac{1}{3})^{3}\alpha^6\Big) \\ &= \frac{1}{6\alpha}\Big(\big(\frac{5}{3}\alpha^{2}\big)^{3} + \frac{1}{27}\alpha^6\Big) \\ &= \frac{1}{6\alpha}\Big(\frac{125}{27}\alpha^{6} + \frac{1}{27}\alpha^6\Big) \\ &= \frac{1}{6}\Big(\frac{126}{27}\alpha^{5}\Big) \\ &= \Big(\frac{21}{27}\alpha^{5}\Big) \\ &= \frac{7}{9}\alpha^{5} \\ \end{align} \]

We can now make one last update our larger overall equation.

\[\begin{align} E^{2}(\alpha,x) &= ||f - \hat{\varphi}(\alpha,x)||^2 \\ & = \int\limits_{x=0}^{\alpha} (-\frac{1}{3}\alpha^2 + 2\alpha x) ^ 2 dx - \frac{34}{45}\alpha^{5} \\ & = \frac{7}{9}\alpha^{5} - \frac{34}{45}\alpha^{5} \\ & = \frac{35}{45}\alpha^{5} - \frac{34}{45}\alpha^{5} \\ & = \frac{1}{45}\alpha^{5} \end{align} \]

That leaves us with

$$ E^{2}(\alpha,x) = \frac{1}{45}\alpha^{5} $$

If we let $\alpha = 3$ our linear approximation function

\[\begin{align} \hat{\varphi}(\alpha,x) & = -\frac{1}{3}\alpha^2 + 2\alpha x \\ \end{align} \]

has an error

\[\begin{align} E^{2}(3,x) &= \frac{1}{45}(3)^{5} \\ &= \frac{1}{5 * (3)^2}(3)^5 \\ &= \frac{1}{5} (3)^3 \\ &= \frac{1 * 27}{5} \\ &= \frac{27}{5} \\ &= 5.4\\ \end{align} \]

If we take the square root…

\[\begin{align} E(3,x) &= \sqrt{\frac{27}{5}} \\ E(3,x) &= \sqrt{5.4} \\ & \approx 2.3 \end{align} \]

That is the smallest error we can have using a line to approximate $2x^2$ for $0 \leq x \leq 3$.