Least Squares - Using a Higher Degree Polynomial

Thomas J. Kennedy

Contents:

Suppose that we want to approximate $f(x) = x$ for the domain $0 \leq x \leq \alpha$ using a polynomial of degree 2… using a continuous approximation.

1 Setting Up

Let us start with what we know:

Since $\hat{\varphi}$ is quadratic… the basis functions are

which leads to…

$$ \begin{align} \hat{\varphi} &= c_0 \pi_1 + c_1 \pi_1 + c_2 \pi_2 \\ &= c_0 + c_1 x + c_2 x^2 \\ \end{align} $$

Since we have three basis functions… $A\vec{c} = \vec{b}$ takes the form…

$$ \left[\begin{array}{ll} \int\limits_{x=0}^{\alpha} \pi_{0} \pi_{0} dx & \int\limits_{x=0}^{\alpha} \pi_{0} \pi_{1}dx & \int\limits_{x=0}^{\alpha} \pi_{0} \pi_{2}dx \\ \int\limits_{x=0}^{\alpha} \pi_{1} \pi_{0} dx & \int\limits_{x=0}^{\alpha} \pi_{1} \pi_{1}dx & \int\limits_{x=0}^{\alpha} \pi_{1} \pi_{2}dx \\ \int\limits_{x=0}^{\alpha} \pi_{2} \pi_{0} dx & \int\limits_{x=0}^{\alpha} \pi_{2} \pi_{1}dx & \int\limits_{x=0}^{\alpha} \pi_{2} \pi_{2}dx \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \int\limits_{x=0}^{\alpha} \pi_0 f dx\\ \int\limits_{x=0}^{\alpha} \pi_1 f dx\\ \int\limits_{x=0}^{\alpha} \pi_2 f dx\\ \end{array}\right] $$

Note how $A$ is a 3-by-3 matrix. We have two constants to compute (i.e., $c_0, c_1, c_2$).

2 Expectations…

Based on the nature of the problem, I suspect that we will end up with

3 Plugging Everything In

We know that

Let us plug everything in.

$$ \left[\begin{array}{ll} \int\limits_{x=0}^{\alpha} dx & \int\limits_{x=0}^{\alpha} x dx & \int\limits_{x=0}^{\alpha} x^2 dx \\ \int\limits_{x=0}^{\alpha} x dx & \int\limits_{x=0}^{\alpha} x^2 dx & \int\limits_{x=0}^{\alpha} x^3 dx \\ \int\limits_{x=0}^{\alpha} x^2 dx & \int\limits_{x=0}^{\alpha} x^3 dx & \int\limits_{x=0}^{\alpha} x^4 dx \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \int\limits_{x=0}^{\alpha} x dx\\ \int\limits_{x=0}^{\alpha} x^2 f dx\\ \int\limits_{x=0}^{\alpha} x^3 f dx\\ \end{array}\right] $$

3.1 Power Rule

Since all integrals take the form

$$ \int\limits_{x=0}^{\alpha} x^n dx \\ $$

We can specialize the definition of the power rule to this problem…

$$ \begin{align} \int\limits_{x=0}^{\alpha} x^n dx &= \frac{1}{n+1} x^{n+1} \Big|_{x=0}^{\alpha} \\ &= \frac{1}{n+1} \left(\alpha^{n+1} - (0)^{n+1}\right) \\ &= \frac{\alpha^{n+1}}{n+1} \\ \end{align} $$

3.2 Using the Extension

Now… we evaluate everything…

$$ \left[\begin{array}{ll} \alpha & \frac{1}{2} \alpha^2 & \frac{1}{3} \alpha^3 \\ \frac{1}{2}\alpha^2 & \frac{1}{3} \alpha^3 & \frac{1}{4} \alpha^4 \\ \frac{1}{3}\alpha^3 & \frac{1}{4} \alpha^4 & \frac{1}{5} \alpha^5 \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{1}{2}\alpha^2 \\ \frac{1}{3}\alpha^3 \\ \frac{1}{4}\alpha^4 \\ \end{array}\right] $$

4 Linear Algebra (Solving)

The next step is to apply some elementary row operations. Let us start with:

$$ \left[\begin{array}{ll} 1 & \frac{1}{2} \alpha & \frac{1}{3} \alpha^2 \\ 1 & \frac{2}{3} \alpha & \frac{1}{2} \alpha^2 \\ 1 & \frac{3}{4} \alpha & \frac{3}{5} \alpha^2 \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{1}{2}\alpha \\ \frac{2}{3}\alpha \\ \frac{3}{4}\alpha \\ \end{array}\right] $$

Let us rewrite the fractions.

$$ \left[\begin{array}{ll} 1 & \frac{6}{12} \alpha & \frac{10}{30} \alpha^2 \\ 1 & \frac{8}{12} \alpha & \frac{15}{30} \alpha^2 \\ 1 & \frac{9}{12} \alpha & \frac{18}{30} \alpha^2 \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{6}{12}\alpha \\ \frac{8}{12}\alpha \\ \frac{9}{12}\alpha \\ \end{array}\right] $$

Now… it is time for

$$ \left[\begin{array}{ll} 1 & \frac{6}{12} \alpha & \frac{10}{30} \alpha^2 \\ 0 & \frac{2}{12} \alpha & \frac{ 5}{30} \alpha^2 \\ 0 & \frac{3}{12} \alpha & \frac{ 8}{30} \alpha^2 \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{6}{12}\alpha \\ \frac{2}{12}\alpha \\ \frac{3}{12}\alpha \\ \end{array}\right] $$

Now… let us scale $\vec{r_1}$ and $\vec{r_2}$

$$ \left[\begin{array}{ll} 1 & \frac{6}{12} \alpha & \frac{10}{30} \alpha^2 \\ 0 & 1 & \alpha \\ 0 & 1 & \frac{32}{30} \alpha \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{6}{12}\alpha \\ 1 \\ 1 \\ \end{array}\right] $$

Let us simplify a few fractions.

$$ \left[\begin{array}{ll} 1 & \frac{1}{2} \alpha & \frac{1}{3} \alpha^2 \\ 0 & 1 & \alpha \\ 0 & 1 & \frac{16}{15} \alpha \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{1}{2}\alpha \\ 1 \\ 1 \\ \end{array}\right] $$

Let us perform $\vec{r_2} - \vec{r_1}$

$$ \left[\begin{array}{ll} 1 & \frac{1}{2} \alpha & \frac{1}{3} \alpha^2 \\ 0 & 1 & \alpha \\ 0 & 0 & \alpha \\ \end{array}\right] \left[\begin{array}{c} c_0 \\ c_1 \\ c_2 \\ \end{array}\right] = \left[\begin{array}{l} \frac{1}{2}\alpha \\ 1 \\ 0 \\ \end{array}\right] $$

We now have an upper-triangular matrix.

5 Switching to Algebra

Let us switch to basic Algebra.

$$ \begin{array}{rrrrrl} (1)c_0 &+& (\frac{1}{2}\alpha)c_1 &+& (\frac{1}{3}\alpha^2)c_2 &=& \frac{1}{2}\alpha \\ (0)c_0 &+& (1)c_1 &+& (\alpha)c_2 &=& 1 \\ (0)c_0 &+& (0)c_1 &+& (\alpha)c_2 &=& 0 \\ \end{array} $$

The third equation simplifies to

$$ c_2 = 0 $$

Now… we can simplify the first two equations.

$$ \begin{array}{rrrrl} (1)c_0 &+& (\frac{1}{2}\alpha)c_1 &=& \frac{1}{2}\alpha \\ (0)c_0 &+& (1)c_1 &=& 1 \\ \end{array} $$

The second equation gives us

$$ c_1 = 1 $$

We are now left with the first equation…

$$ \begin{eqnarray} (1)c_0 + (\frac{1}{2}\alpha)c_1 &=& \frac{1}{2}\alpha \\ c_0 + (\frac{1}{2}\alpha)(1) &=& \frac{1}{2}\alpha \\ c_0 + (\frac{1}{2}\alpha) &=& \frac{1}{2}\alpha \\ c_0 &=& -(\frac{1}{2}\alpha) + \frac{1}{2}\alpha \\ c_0 &=& 0 \\ \end{eqnarray} $$

6 The Result

Now that we have $c_0$, $c_1$, and $c_2$ the approximation function can be written as…

$$ \hat{\varphi} = (0) 1 + (1) x + (0) x^2 $$

or in its simplified form as…

$$ \hat{\varphi} = x $$

7 What is the Error

Now… for the important question: How “good” is our approximation?

$$ f(x) = \hat{\varphi}(x,\alpha) \Rightarrow E^2(x, \alpha) = 0 $$

Since we ended up with our orignal function… the error is zero!