1 Some Simple Examples

1.1 Diagonal Matrix

This block of code might be found in an early step in a linear algebra application. It sets up an NxN matrix with 1.0 on the diagonals and 0.0 on all of the off-diagonal elements:

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
       if (i == j)
         matrix[i][j] = 1.0;   ➀
       else
         matrix[i][j] = 0.0;   ➁
    }
}

The most deeply nested statements in this code are ➀ and ➁. Looking at them, we can see that they involve address calculations for the array indexing, and assignment of a single double. All of those calculations are $O(1)$.

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    matrix[i] = new double[N];
    for (int j = 0; j < N; ++j)
    {
       if (i == j)
         matrix[i][j] = 1.0;   // O(1)
       else
         matrix[i][j] = 0.0;   // O(1)
    }
}

The if statement condition is $O(1)$,

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
       if (i == j)         // cond: O(1)
         matrix[i][j] = 1.0;   // O(1)
       else
         matrix[i][j] = 0.0;   // O(1)
    }
}

which means that

\[ \begin{align} t_{\mbox{if}} &= t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \\ &= O(1) + \max(O(1), O(1)) \\ &= O(1) \end{align} \]

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
       if (i == j)         // cond: O(1)  total: O(1)
         matrix[i][j] = 1.0;   // O(1)
       else
         matrix[i][j] = 0.0;   // O(1)
    }
}

Collapsing,

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
       // O(1)
    }
}

The inner for loop has initialization, condition, and increment all O(1). It executes N times.

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    // init: O(1)
    for (int j = 0; j < N; ++j) // cond: O(1)  incr: O(1)  #: N
    {
       // O(1)
    }
}

So by our rule for for loops,

\[\begin{align} t_{\mbox{for}} &= t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= O(1) + \sum_{j=0}^{N-1}(O(1) + O(1) + O(1)) + O(1) \\ &= O(1) + N * (O(1) + O(1) + O(1)) + O(1) \\ &= O(1) + N * (O(1)) + O(1) \\ &= O(1) + O(N) + O(1) \\ &= O(N) \end{align}\]

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    // init: O(1)
    for (int j = 0; j < N; ++j) // cond: O(1)  incr: O(1)  #: N  total: O(N)
    {
       // O(1)
    }
}

Collapsing:

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    // init: O(1)
    // O(N)
}

The statement sequence making up this function body adds up to $O(N)$

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i)
{
    // O(N)
}

The remaining for loop has initialization, condition, and increment all O(1). It executes N times.

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i) // cond: O(1)  incr: O(1)  #: N
{
    // O(N)
}

And so the time for this loop is

\[\begin{align} t_{\mbox{for}} &= t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= O(1) + \sum_{i=0}^{N-1}(O(1) + O(1) + O(N)) + O(1) \\ &= O(1) + N * (O(1) + O(1) + O(N)) + O(1) \\ &= O(1) + N * (O(N)) + O(1) \\ &= O(1) + O(N^2) + O(1) \\ &= O(N^2) \end{align}\]

double[][] matrix = new double[N][N];
for (int i = 0; i < N; ++i) // cond: O(1)  incr: O(1)  #: N  total: O(N^2)
{
    // O(N)
}

Collapsing:

double[][] matrix = new double[N][N];
// O(N^2)

The first statement allocates a 2D array of double, containing a total of $N^2$ elements. Java initializes each of those $N^2$ elements to zero. Because it must visit each of those $N^2$ elements to do so, this is $O(N^2)$.

double[][] matrix = new double[N][N]; // O(N^2)
// O(N^2)

and the entire statement sequence adds up to $O(N^2)$.

There are a few things that I hope you take notice of here:

1.2 Working with a Matrix

Continuing with our NxN matrix, suppose that later in the code we saw:

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];
        matrix[j][i] += matrix[i][i];
        if (i == j)
        {
            for (int k = 0; k < N; ++k)
                matrix[i][i] *= 2.0;
        }
    }
}

Here we have three nested loops, each apparently repeating N times, so we might guess that this will be $O(N^3)$. let’s see if this guess holds up.

Again, we can start by marking simple statements that involve only arithmetic and assignment of primitive types.

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)
        {
            for (int k = 0; k < N; ++k)
                matrix[i][i] *= 2.0;      // O(1)
        }
    }
}

Looking at the innermost loop,

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)
        {
            for (int k = 0; k < N; ++k)   //cond: O(1)  iter:    O(1) #: N
                matrix[i][i] *= 2.0;      // O(1)
        }
    }
}

and the loop time is

\[\begin{align} t_{\mbox{for}} &= t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= O(1) + \sum_{k=0}^{N-1}(O(1) + O(1) + O(1)) + O(1) \\ &= O(1) + N * (O(1) + O(1) + O(1)) + O(1) \\ &= O(1) + N * (O(1)) + O(1) \\ &= O(1) + O(N) + O(1) \\ &= O(N) \end{align}\]

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)
        {
            for (int k = 0; k < N; ++k)   //cond: O(1)  iter: O(1) #: N  total: O(N)
                matrix[i][i] *= 2.0;      // O(1)
        }
    }
}

Collapsing:

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)
        {
           // O(N)
        }
    }
}

The if condition is $O(1)$.

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)  // cond: O(1)
        {
           // O(N)
        }
    }
}

And we can then evaluate the if as

\[ \begin{align} t_{\mbox{if}} &= t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \\ &= O(1) + \max(O(N), O(1)) \\ &= O(N) \end{align} \]

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)
    {
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)  // cond: O(1)  total: O(N)
        {
           // O(N)
        }
    }
}

Now, commonly we would, at this point, collapse the if code and just retain the total complexity. But there are three things that hint to me that this might not be the best approach:

1. The if is inside a loop.
2. The then and else part complexities are different, and
3. The if condition seems highly selective (will take one path much more often than the other).

Whenever I see those three characteristics together, I think it’s worth asking whether the more expensive of the two if parts gets executed often enough, in the worst case, to actually dominate the overall loop.

Ignore the outermost loop for the moment.

    for (int j = 0; j < N; ++j)
    {// O(N)
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)  // cond: O(1)  total: O(N)
        {
           // O(N)
        }
    }

Question: How many times, total, is the if statement going to be executed by this loop?

Question: How many times, total, will the if statement take the “then” (true) branch?

Question: How many times, total, will the if statement take the “else” (false) branch?

OK, now the rest of that loop is easy enough to handle:

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)  // cond: O(1)  iter: O(1)  #: N
    {// O(N)
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)  // cond: O(1)  total: O(N)
        {
           // O(N)
        }
    }
}

so that

\[\begin{align} t_{\mbox{for}} &= t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= O(1) + \sum_{j=0}^{N-1}(O(1) + O(1) + t_{\mbox{body}}) + O(1) \\ &= O(1) + \sum_{j=0}^{N-1}(O(1) + O(1)) + \sum_{j=0}^{N-1} t_{\mbox{body}} + O(1) \\ &= O(1) + N * O(1) + \sum_{j=0}^{N-1} t_{\mbox{body}} \\ &= O(1) + O(N) + \sum_{j=0}^{N-1} t_{\mbox{body}} \\ &= O(N) + \sum_{j=0}^{N-1} t_{\mbox{body}} \\ \end{align}\]

OK, now let’s think about what our earlier questions told us about the sum of the loop body times over all of the iterations. That sum will add together a total of N executions of the body. Exactly one of those will take $O(N)$ time and $N-1$ of those will take $O(1)$ time. So

\[\begin{align} t_{\mbox{for}} &= O(N) + \sum_{j=0}^{N-1} t_{\mbox{body}} \\ &= O(N) + O(N) + (N-1)O(1) \\ &= O(N) + O(N) + O(N) - O(1) \\ &= O(N) \\ \end{align}\]

So what we can see is that the expensive option of the if is done so rarely that it does not dominate the rest of the sum – it kind of disappears into the overall sum.

for (int i = 0; i < N; ++i)
{
    for (int j = 0; j < N; ++j)  // cond: O(1)  iter: O(1)  #: N  total: O(N)
    { // O(N)
        matrix[i][j] += matrix[i][i];     // O(1)
        matrix[j][i] += matrix[i][i];     // O(1)
        if (i == j)  // cond: O(1)  total: O(N)
        {
           // O(N)
        }
    }
}

Now we can collapse that “j” loop:

for (int i = 0; i < N; ++i)
{
    // O(N)
}

The final loop follows the familiar form:

for (int i = 0; i < N; ++i)  // cond: O(1)  iter: O(1)  #: N
{
    // O(N)
}

\[\begin{align} t_{\mbox{for}} &= t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= O(1) + \sum_{i=0}^{N-1}(O(1) + O(1) + O(N)) + O(1) \\ &= O(1) + N * (O(1) + O(1) + O(N)) + O(1) \\ &= O(1) + N * (O(1)) + O(1) \\ &= O(1) + O(N^2) + O(1) \\ &= O(N^2) \end{align}\]

And the entire block of code is $O(N^2)$.

That is not what we predicted by just looking at the depth of nesting of the loops. Surprises demand explanation. In this case we have the explanation: the innermost loop is executed only (1/N)th of the time, so we wind up being a factor of $N$ faster than we had originally predicted.

2 Ordered Insertion

​
We have worked with this function in earlier lessons:

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          ➀
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  ➁
            intoArray[i] = intoArray[i-1];                     ➂
            --i;
        }
        intoArray[i] = value;                                  ➃
        return i;
    }  

• We start from the high end of the array (➀) and check to see if that’s where we want to insert the data (➁).

  • If so, fine. We exit from the loop.
  • If not, we move the preceding element up one (➂) and then check to see if we want to insert value into the “hole” left behind. We repeat this step as necessary.

• Once we exit from the loop (➃), we insert the value into the hole at our chosen position.

2.1 Analysis of insertInOrder

We start our analysis with the easy stuff – mark all of the non-compound statements.

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          // O(1)
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  
            intoArray[i] = intoArray[i-1];                     // O(1)
            --i;                                               // O(1)
        }
        intoArray[i] = value;                                  // O(1)
        return i;                                              // O(1)
    }  

Next, looking at the while loop, what is the complexity of the loop condition?

When we have an expression that involves a function call, we need to know the complexity of that function.

But the compareTo function here could be anything. Literally, anything. That’s because this insertInOrder function is a generic and could be used with any data type that supports the Comparable interface, including future data types that no one has yet written.

So we can’t say how long this function takes to execute. We’ll just have to introduce a symbol, say, $t_c$ to denote the time taken by the compareTo function.

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          // O(1)
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  // cond: O(t_c)
            intoArray[i] = intoArray[i-1];                     // O(1)
            --i;                                               // O(1)
        }
        intoArray[i] = value;                                  // O(1)
        return i;                                              // O(1)
    }  

This is a pretty common occurrence when analyzing generics.

Moving on, we see that the loop repeats at most size times.

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          // O(1)
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  // cond: O(t_c) # size
            intoArray[i] = intoArray[i-1];                     // O(1)
            --i;                                               // O(1)
        }
        intoArray[i] = value;                                  // O(1)
        return i;                                              // O(1)
    }  

The loop body is $O(1)$, so

\[ \begin{align} t_{\mbox{while}} &= \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ &= \sum_{i=1}^{\mbox{size}} (O(t_c) + O(1)) + O(1) \\ &= O\left(\sum_{i=1}^{\mbox{size}} t_c\right) + O(1) \\ &= O(\mbox{size * t_c}) + O(1) \\ &= O(\mbox{size * t_c}) \end{align} \]

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          // O(1)
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  // cond: O(t_c) # size total: O(size* t_c)
            intoArray[i] = intoArray[i-1];                     // O(1)
            --i;                                               // O(1)
        }
        intoArray[i] = value;                                  // O(1)
        return i;                                              // O(1)
    }  

Replacing the loop by this quantity … ,

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          // O(1)
        // O(size * t_c)
            intoArray[i] = intoArray[i-1];                     // O(1)
            --i;                                               // O(1)
        }
        intoArray[i] = value;                                  // O(1)
        return i;                                              // O(1)
    }  

… we get down to a statement sequence composed of $O(1)$ and $O(\mbox{size})$ terms. The total run time is therefore

\[ t_{\mbox{addInOrder}} = O(1) + O(\mbox{size * t_c}) + O(1) + O(1) + O(1) + O(1) = O(\mbox{size * t_c}) \]

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        // O(size * t_c)
    }  

So, we conclude that this function is in $O()\mbox{size} * t_c)$, where $t_c$ is the worst case time of the compareTo function.

Note that, because none of the inputs to this function are actually named “n”, it is not proper to say the function is “O(n)” unless we explicitly define “n” to be equal to size * $t_c$.

​

2.2 Special Case Behavior

    public static <T extends Comparable<T>> int insertInOrder(T value, T[] intoArray, int size) {
        int i = size;                                          
        while(i > 0 && value.compareTo(intoArray[i-1]) < 0) {  
            intoArray[i] = intoArray[i-1];                     
            --i;                                               
        }
        intoArray[i] = value;                                  
        return i;                                              
    }  

A special point worth noting:

• If we are adding a value that is greater than all elements already in intoArray, this algorithm does 0 iterations of the loop.

• Suppose we are given a series of values to insert into an initially empty array, and that these values are already sorted.

  • Then each new value will be greater than all the ones already inserted into the array.

  • Each call to insertInOrder will use 0 iterations.

  • and so each call runs in $O(t_c)$ time (for this special case of inserting sorted elements)

• We’ll make use of this special case later when we incorporate this function into more complex algorithms.

3 Sequential Search

Another common utility function is to search through a range of data looking for a “search key”, returning the position where it is found and some “impossible” position if it cannot be found.

This code is from chapter 7 of the text:

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  for (int i=0; i<A.length; i++) // For each element
    if (A[i] == K)               // if we found it
       return i;                 //   return this position
  return A.length;               // Otherwise, return the array length
}

3.1 Analysis

Again, we can start the analysis by marking the “easy” stuff – the non-compound statements.

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  for (int i=0; i<A.length; i++) // 
    if (A[i] == K)               //
       return i;                 // O(1)
  return A.length;               // O(1)
}

Looking at the if statement, the condition is $O(1)$.

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  for (int i=0; i<A.length; i++) // 
    if (A[i] == K)               // cond: O(1)
       return i;                 // O(1)
  return A.length;               // O(1)
}

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  for (int i=0; i<A.length; i++) // 
    if (A[i] == K)               // cond: O(1) total: O(1)
       return i;                 // O(1)
  return A.length;               // O(1)
}

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  // O(1)
  for (int i=0; i<A.length; i++) // cond: O(1) #: A.length
    if (A[i] == K)               // cond: O(1) total: O(1)
       return i;                 // O(1)
  return A.length;               // O(1)
}

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  // O(1)
  for (int i=0; i<A.length; i++) // cond: O(1) #: A.length total: O(A.length)
    if (A[i] == K)               // cond: O(1) total: O(1)
       return i;                 // O(1)
  return A.length;               // O(1)
}