2 Setting Up

We know the definitions of $A$ (a.k.a. $X^{T}X$) and $\vec{b}$ (a.k.a. $X^{T}Y$).

$$ A = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)\pi_{0}(x_i) & \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)\pi_{1}(x_i) \\ \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)\pi_{0}(x_i) & \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)\pi_{1}(x_i) \\ \end{array}\right] $$

and

$$ \vec{b} = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)f(x_i) \\ \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)f(x_i) \\ \end{array}\right] $$

Since we are working with linear approximation functions… we know that:

• $\pi_0 = 1$
• $\pi_1 = x$

We can now go from…

$$ A\vec{c} = \vec{b} $$

to

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)\pi_{0}(x_i) & \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)\pi_{1}(x_i) \\ \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)\pi_{0}(x_i) & \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)\pi_{1}(x_i) \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}\pi_{0}(x_i)f(x_i) \\ \sum\limits_{i=0}^{k-1}\pi_{1}(x_i)f(x_i) \\ \end{array}\right] $$

If we apply the definitions of $\pi_0$ and $\pi_1$… the expression becomes…

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}(1)(1) & \sum\limits_{i=0}^{k-1}(1)(x_i) \\ \sum\limits_{i=0}^{k-1}(x_i)(1) & \sum\limits_{i=0}^{k-1}(x_i)(x_i) \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}(1)f(x_i) \\ \sum\limits_{i=0}^{k-1}(x_i)f(x_i) \\ \end{array}\right] $$

Let us simplify a few of the products.

$$ \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1} 1 & \sum\limits_{i=0}^{k-1} x_i \\ \sum\limits_{i=0}^{k-1} x_i & \sum\limits_{i=0}^{k-1}(x_i)^{2} \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}f(x_i) \\ \sum\limits_{i=0}^{k-1}x_i * f(x_i) \\ \end{array}\right] $$

The first sum is just the number of points (i.e., $k$).

$$ \left[\begin{array}{rr} k & \sum\limits_{i=0}^{k-1} x_i \\ \sum\limits_{i=0}^{k-1} x_i & \sum\limits_{i=0}^{k-1}(x_i)^{2} \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}f(x_i) \\ \sum\limits_{i=0}^{k-1}x_i * f(x_i) \\ \end{array}\right] $$

3 Making Writing More Convenient

Before we start solving for $c_0$ and $c_1$… let us replace the sums with placeholder variables (imagine writing each sum a dozen or so times by hand).

• $S_{x_i} = \sum\limits_{i=0}^{k-1} x_i$
• $S_{x_i^2} = \sum\limits_{i=0}^{k-1}(x_i)^{2}$

$$ \left[\begin{array}{rr} k & S_{x_i} \\ S_{x_i} & S_{x_i^{2}} \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \sum\limits_{i=0}^{k-1}f(x_i) \\ \sum\limits_{i=0}^{k-1}x_i * f(x_i) \\ \end{array}\right] $$

4 Solving the System

Let us start with the first two elementary row operations.

1. $\frac{1}{k} \vec{r_0}$
2. $\frac{1}{S_{x_i}} \vec{r_1}$

$$ \left[\begin{array}{rr} 1 & \frac{1}{k} S_{x_i} \\ 1 & \frac{1}{S_{x_i}} S_{x_i^{2}} \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \frac{1}{k} \sum\limits_{i=0}^{k-1}f(x_i) \\ \frac{1}{S_{x_i}} \sum\limits_{i=0}^{k-1}x_i * f(x_i) \\ \end{array}\right] $$

We will perform one elimination step before switching to a vanilla algebra notation.

$$ \left[\begin{array}{rr} 1 & \frac{1}{k} S_{x_i} \\ 0 & \frac{1}{S_{x_i}} S_{x_i^{2}} - \frac{1}{k} S_{x_i} \\ \end{array}\right] \left[\begin{array}{rr} c_0 \\ c_1 \\ \end{array}\right] = \left[\begin{array}{rr} \frac{1}{k} \sum\limits_{i=0}^{k-1}f(x_i) \\ \frac{1}{S_{x_i}} \sum\limits_{i=0}^{k-1}x_i * f(x_i) - \frac{1}{k} \sum\limits_{i=0}^{k-1}f(x_i)\\ \end{array}\right] $$

We can rewrite the last row as the algebraic expression it represents…

$$ \begin{eqnarray} c_1 \left(\frac{1}{S_{x_i}} S_{x_i^{2}} - \frac{1}{k} S_{x_i}\right) &=& \frac{1}{S_{x_i}} \sum\limits_{i=0}^{k-1}x_i * f(x_i) - \frac{1}{k} \sum\limits_{i=0}^{k-1}f(x_i)\\ c_1 \left(\frac{S_{x_i}}{S_{x_i}}S_{x_i^{2}} - \frac{S_{x_i}}{k} S_{x_i}\right) &=& \frac{S_{x_i}}{S_{x_i}} \sum\limits_{i=0}^{k-1}x_i * f(x_i) - \frac{S_{x_i}}{k} \sum\limits_{i=0}^{k-1}f(x_i)\\ c_1 \left(S_{x_i^{2}} - \frac{1}{k} \left(S_{x_i}\right)^{2}\right) &=& \sum\limits_{i=0}^{k-1}x_i * f(x_i) - \frac{S_{x_i}}{k} \sum\limits_{i=0}^{k-1}f(x_i)\\ c_1 \left(kS_{x_i^{2}} - \frac{k}{k} \left(S_{x_i}\right)^{2}\right) &=& k\sum\limits_{i=0}^{k-1}x_i * f(x_i) - k\frac{S_{x_i}}{k} \sum\limits_{i=0}^{k-1}f(x_i)\\ c_1 \left(kS_{x_i^{2}} - \left(S_{x_i}\right)^{2}\right) &=& k\sum\limits_{i=0}^{k-1}x_i * f(x_i) - S_{x_i} \sum\limits_{i=0}^{k-1}f(x_i)\\ c_1 &=& \frac{k\sum\limits_{i=0}^{k-1}x_i * f(x_i) - S_{x_i} \sum\limits_{i=0}^{k-1}f(x_i)} {kS_{x_i^{2}} - \left(S_{x_i}\right)^{2}} \\ \end{eqnarray} $$

I have no desire to ever perform any of that simplification within a matrix!

5 Taking Stock…

We now have the slope ($c_1$).

$$ c_1 = \frac{k\sum\limits_{i=0}^{k-1}x_i * f(x_i) - S_{x_i} \sum\limits_{i=0}^{k-1}f(x_i)} {kS_{x_i^{2}} - \left(S_{x_i}\right)^{2}} \\ $$

Let us now take the first row of the matrix… and rewrite it algebraically.

$$ \begin{eqnarray} c_0 + c_1 \frac{1}{k} S_{x_i} &=& \frac{1}{k} \sum\limits_{i=0}^{k-1}f(x_i) \\ c_0 &=& \frac{1}{k} \left(\sum\limits_{i=0}^{k-1}f(x_i) - c_1 S_{x_i}\right) \\ \end{eqnarray} $$

We could plug in $c_1$ and simplify further, but I prefer solving for $c_1$ first and using that result to compute $c_0$ After all… that is probably the approach used in software.

6 The Actual Final Form

Using the definitions of the $S$ terms…

$$ \begin{eqnarray} c_1 &=& \frac{ k\sum\limits_{i=0}^{k-1}x_i * f(x_i) - \left(\sum\limits_{i=0}^{k-1} x_i\right) \sum\limits_{i=0}^{k-1}f(x_i) } { k\left(\sum\limits_{i=0}^{k-1} x_{i}^{2}\right) - \left(\sum\limits_{i=0}^{k-1} x_i\right)^{2} } \\ \end{eqnarray} $$

and

$$ \begin{eqnarray} c_0 &=& \frac{1}{k} \left(\sum\limits_{i=0}^{k-1}f(x_i) - c_1 \left(\sum\limits_{i=0}^{k-1} x_i\right)\right) \\ \end{eqnarray} $$