1.2 Simplifying Rules

1.3 In search of a method

The textbook stops short, in my opinion, of giving a practical approach to actually doing the big-O analysis. That’s what we will do in the remainder of this reading.

We will be looking at two things:

1. How do we derive the worst-case expression for each kind of programming language construct?

2. How can we annotate our code to express those worst-case behaviors in a way that amounts to a proof of your results?

A worst-case analysis is really a kind of mathematical proof, and people’s styles of writing mathematical proofs vary so much that the results are often unreadable by anyone other than the author.

So in this lesson I am going to show you my own copy-and-paste style for conducting and documenting a worst-case analysis. This is the technique that I will use through this course whenever I am giving an analysis. And it is also the technique that I will expect you to use when you do assignments in which I ask you to show me your work on the algorithm analysis.

This technique takes advantage of the fact that these days we don’t usually write down our algorithms using paper and pencil. Instead we use text editors which are very good in doing copy and paste. Consequently the technique relies on your ability to make copies of an algorithm over and over again, making small changes to the algorithm at each stage that reflect the analysis that you have actually done.

2 Some Overall Guidelines

First, some basic observations.

2.1 It All Boils Down to Addition

If you want to know how much time a complicated process takes, you figure that out by adding up the times of its various components.

Now, we will have a lot of situations where we use multiplication and other operations, but those are always “special cases”. For example, if I tell you that a certain component of a program takes time $t_0$ and we later discover that this component is inside a loop that repeats 1000 times, we may write the total as $1000 t_0$, but that’s only because

\[ 1000 t_0 = \sum_{i=1}^{1000} t_0 = t_0 + t_0 + \ldots (997 \mbox{ times})\ldots + t_0 \]

It’s just a simplification for a lot of additions.

When in doubt, just add things up.

2.2 Nothing Happens in O(0) Time

Even doing nothing takes time. “Not doing” a block of code still requires at least a clock cycle to move the address counter forward to the next block of code. So, even doing nothing is $O(1)$, not $O(0)$.

So if your math comes out to zero (or even worse, negative time values), it’s time to go back and check your math.

2.3 All of Your Variables Must be Defined

In math, as in programming, all of your variables must be declared/defined before you can use them.

If you reach a conclusion that a certain Java function is $O(N + M^2)$, then $N$ and $M$ have to actually mean something. That meaning may come because they are actual variables in the code:

void foo(double[] array, int N, int M) // O(N + M^2)

or because, in the style of mathematical proofs everywhere, we have supplied our own definition, e.g.,

void foo (vector<double> v)
Let N be the number of elements in the vector v and let M
be the number of those elements that are greater than 1.0.

But, one way or another, every symbol in your big-O expressions needs to be properly declared.

• Don’t fall in love with “n”. Even though our definition of complexity talks about $n$, that doesn’t mean that there has to be a variable named “n” in the code, or that “n” means anything at all in a specific analysis problem.
• And, don’t forget that , in mathematics as in Java, n and N are not the same variable.

The possibility that we may use variables from the code in our mathematical description of complexity leads to a closely related rule:

2.4 Complexity is Written in Terms of the Inputs

The complexity of a block of code must be a function of the inputs (only!) to that block.

If you are looking at a block of code like this:

int x = y + z;
int k = foo (x, y, z);
bar (k);

your big-O expression to describe this code cannot involve $k$. That’s because $k$ does not even exist when we start the block of code. And, in many cases, e.g.:

void baz(int y, int z)
{
    int x = y + z;
    int k = foo (x, y, z);
    bar (k);
}

$k$ will not exist after we have completed the block of code. In this case, because $k$ is computed from x, y, and z, and x was computed from y and z, we would need to say, instead of $O(k)$, the complexity could be $O(\mbox{foo}(y+z,y,z))$.

2.5 The Complexity of Any Block of Code Must be Numeric

Our definition of $t \in O(f(\bar{n}))$ says that “… $t < c*f(n)$…”.

The “<” and multiplication by a constant $c$ only makes sense if $f(\bar{n})$ is a numeric quantity of some kind. So if we were looking at

void foo (vector<string> v, string str)

it might be valid to say that the complexity of foo is O(1), O(v.size()), O(N) where N is the number of vowels in str, etc., because all of those are numeric quantities. But you cannot possibly say the complexity is O(v) or O(str) because v and str are not numeric. They can’t be multiplied by a constant c and cannot be compared to a numeric time value.

2.6 Surprises Demand Explanation

In many scientific and engineering fields, students are taught that any complicated calculations should be subjected to “sanity checks” that may

• check parts of the calculation for internal consistency

  • Many of the rules we’ve already been discussing in this section would fall into this category

• compare the answer against a simpler, rough calculation (often called a “back-of-the-envelope” method, after the stereotypical image of an engineer scratching a few numbers on a handy piece of paper and suddenly announcing an approximate answer to the problem that everyone else has been struggling with).

Back-of-the-envelope calculations are not a substitute for doing the “real” problem solution, but they do provide an important function as a sanity check. It’s entirely possible that, when we do the real calculation, we will come up with a different answer than the back-of-the-envelope answer. If so, it’s worthwhile to take a moment to understand why the two answers are different.

1. Maybe we made a mistake in one computation or the other. If so, we need to find that mistake and correct it.
2. On the other hand, maybe we did everything correctly, and there is some deep feature of this problem that isn’t obvious to a quick back-of-the-envelope approach, something that accounts for the surprise of the final result of the full calculation.

   If so, we need to identify and explain that surprise before we can have full confidence in our final answer.

Example 1: A back-of-the-Envelope method for loops

For example, most of the time of typical code is spent in loops, and so loops tend to dominate the worst-case complexity. A reasonable back-of-the-envelope technique is to

• Look at the most deeply nested loops in the algorithm. If the loops are nested $k$ deep, and each appears to execute up to $N$ times, then $N^k$ is a reasonable first guess for the algorithm complexity.

It’s only a guess, but if you come up with something else, you should be sure that you understand where the difference is coming from.

3 The Complexity of Expression Evaluation

Most of the actual executable part of a typical program is made up of expression evaluation. Even many things that we refer to as statements, such as assignment statements or procedure/function calls, e.g.

x = y + z;
doSomethingWith(x,y,z);  

are just larger expressions in Java.

3.1 Arithmetic & Relational Operations on Primitive Types are O(1)

If we restrict our attention to the primitive types (integers, floating point numbers, booleans, characters, pointers, and references, then all operators provided by the language are O(1).

Think about something like:

int x = ...
int y = ...
  ⋮
x + y   // What is the complexity of this?  

the time required to compute the sum is independent of how large the values in x and y might be. It takes no more time to compute 1000000 + 987654321 that it takes to compute 0 + 1. It might take a bit longer to add a pair of 64-bit long integers than a pair of 8-bit short integers, but the available bit widths is fixed by the environment (CPU and compiler) and, for each available choice of widths, is a constant no matter what the actual numbers involved might be.

The same is true for subtraction, multiplication, boolean operations, relational operators, etc.

When we combine these operators into more complicated expressions, we are simply instructing the machine to do one of these operations, then another, and so on. For example, if we had integer variable x, y, and z, then an expression like x + y*z is $O(1)$. There are two operations in that expression, a multiplication and an addition, and those operations each run in $O(1)$ time (i.e., they each take a constant amount of time regardless of the amount of data being manipulated by the program). So the expression has complexity $O(c_+ + c_*)$, which, by our rules for big-O algebra, simplifies to $O(1)$.

On the other hand the complexity of an expression like foo(i) + bar(j) would be the sum of the complexities of the calls to foo and to bar. We’ll talk about how those calls are evaluated shortly.

Question: What is the worst-case complexity of the expression `(x < 100) & (y + 1 > 0)``?

3.2 Assignments are O(1)

Really, assignments of any data type that has a fixed number of bytes (ints, doubles, chars, etc., but not strings, vectors, etc.) are O(1).

The explanation is that it takes the same amount of time to copy a byte that contains 127 as a byte that contains 0. The actual value stored there is irrelevant. The number of bytes making up the total does matter, but if that is fixed, then copying that number of bytes takes the same amount of time for all data types of that size.

For non-primitive types in Java, assignment is just copying a reference/pointer. Again, there area fixed number of bytes to be copied, and the time does not depend on how large the addresses are stored within.

3.3 Basic Address Calculations are O(1)

Array indexing is just an address calculation. To access an array element a[i], the compiler generated code that does the equivalent of

address = address-of-a + (i * size-of-an-element

Basically, it boils down to an integer multiplication and addition. So

• Array indexing a[i] is $O(1)$.

Similarly,

• Class member selection myObject.member is $O(1)$. The ‘.’ is just an address calculation in which the base address of the entire myObject structure is added together with a compiler-assigned integer offset for the beginning of member within each structure of that same type. It’s just an addition.

3.4 Function Calls

So far, everything we might write into an expression has been pretty quick. That stops once we start to consider function calls (including programmer-supplied operator overloads).

Addition is still king. When we have an expression like

    int k = foo (x, y, z);

we still add the complexity of the function call to the complexity of the other operations (in this case, integer assignment). But what is the complexity of the function call?

To answer that, someone needs to have analyzed the body of that function and figured out its complexity.

• Sometimes someone else will have already done that for us. In that case we can simply look it up.

• If no one else has done the job for us, we will have to set our current problem aside for the moment and analyze the function body ourselves. A fair number of functions form the Java library are so simple that, by a few weeks from now, you’ll be able to analyze them at a glance. Other functions may require a substantial effort.

For now, let’s assume that, one way or the other, we have found the complexity of a function that we are calling. What does that mean? What will we actually “know”?

3.5 Annotating Assignment Statements

As promised at the beginning of this section, we will be recording our conclusions about the complexity of the various components of the code by annotating the code itself with comments.

For statement-level expressions (e.g., assignments or void function calls) the annotation is very simple. We simply write the big-O expression at the end of the line as a // comment. For example:

       int[] array = new int[100]; // // O(1)
          ⋮
       a[i] = a[i-1];  // O(1)

and

       String array = new String[100]; // O(1)
          ⋮
       for (int i = N; i > 0, --i)
           a[i] = a[i-1];  // O(1)

Now, typing mathematics in plain text like this has some limitations that we will need to work around.

• The most common is the need to use superscripts to denote raising something to a power. We will use the caret (^) character for that purpose, for example, typing $O(N^2)$ as O(N^2).

  Not only does ^ suggest raising something up, but in some programming languages, it is actually used as the “raise to a power” operator.

• Less often, we will want to denote subscripts. We will use the underscore (_) character for that purpose. For example, we would type $O(x_0 + x_1)$ as O(x_0 + x_1).

As we conduct an analysis of complexity, we will wind up with these kinds of annotations sprinkled throughout the code, e.g.,

int <T> seqOrderedSearch(const Comparable<T> list[], int listLength, 
               Comparable<T> key)
{
    int loc = 0;    // O(1) 

    while (loc < listLength && list[loc].compareTo(key) < 0)
      {
       ++loc;      // O(1) 
      }
    return loc;   // O(1) 
}

(A return statement is essentially a copy or assignment to a special variable in the caller, so, like assignment, it will be $O(1)$.)

4 The Complexity of Compound Statements

Of course, expression evaluation and assignment may account for the bulk of single statements in a program, but what allows us to combine single statements into a full algorithm is the idea of compound statements: sequences (a.k.a. blocks), conditionals (e.g., if), and loops.

Compound statements are single statements that are built around one or more simpler component statements. For example, a while loop is build around a loop body, which is itself a statement. Some of the components of a compound statement may also be smaller compound statements, as when we nest loops inside one another.

There is a general observation we can make about analyzing compound statements. In order to compute an upper bound on the time required by a compound statement, we will almost always need to know the bounds on the times of its components. Consequently

We usually analyze compound statements in an inside-out fashion, starting with the most deeply nested components and working our way outward from there.

4.1 Sequences of Statements

The simplest form of compound statement is the sequence or block, written in Java between { } brackets. Examples include function bodies, loop bodies, and the “then” and “else” parts of if statements.

A sequence tells the machine to process statements one after the other. So,

The time for a sequence of statements is the sum of the times of the individual statements.

More formally, for a sequence of statements $s_1, s_2, …, s_k$

\[ t_{\mbox{seq}} = \sum_{i=1}^k t_{s_i} \]

This rule applies when we have a block of statements in a straight line – no way to jump into the middle of the block, no way to jump out. When we have sequential arrangements of statements like that, then the complexity of the sequence as a whole is sum of the complexities of the statements

Example 2: Statement Sequence

{
  a[10*i+j] = foo(i,j);
  bar(a[10*i+j], i, j);
}

In the code shown here, if I look at the first assignment statement, its complexity depends on the complexity of the body of the function foo.

Let’s just say for the sake of example that the function foo(x,y) has a body of complexity $O(x^{2})$. Substituting the actual parameter i from the call for the formal parameter x, we conclude that the first statement has complexity $O(i^{2})$

We annotate the first statement accordingly:

{
  a[10*i+j] = foo(i,j); // O(i^2)
  bar(a[10*i+j], i, j);
}

The second statement involves a call to bar. For the sake of example, let’s assume that bar(x,y,z) has complexity of $O(y+z)$. Again, to get the complexity of the call we must substitute the actual parameters for the formals, and doing so indicates that the second statement has complexity $O(i + j)$.

We would record our conclusion by annotating that statement:

{
  a[10*i+j] = foo(i,j); // O(i^2)
  bar(a[10*i+j], i, j); // O(i + j)
}

That would mean that the complexity for the sequence of statements consisting these two individual statements would be sum of those two complexities which would be $O(i^{2}+i+j)$ and since $i^{2}$ is greater than $i$, this simplifies to $O(i^{2} + j)$.

4.1.1 Annotating Statement Sequences

In some cases we won’t bother annotating the total complexity of a statement sequence, particularly when it is short or the answer is obvious (e.g., because all of the component statements have the same complexity).

When we do want to record the total for a statement sequence, we will do so by adding an annotation // total: O(...) at the top of the sequence.

In our example above, we would record our conclusion like this:

{ // total: O(i^2 + j)
  a(10*i+j) = foo(i,j); // O(i^2)
  bar(a(10*i+j), i, j); // O(i + j)
}

As we proceed through a proof, we may decide that we don’t need the details inside the sequence any more. In that case, we may opt to make a separate copy of the code, replace the entire sequence with its total complexity, and add a brief text in between explaining what we are doing, e.g.,

{ // total: O(i^2 + j)
  a(10*i+j) = foo(i,j); // O(i^2)
  bar(a(10*i+j), i, j); // O(i + j)
}

Collapsing that block, 

{
  // O(i^2 + j)
}

4.2 Conditional statements

When we have an if statement, we know that we will execute either the then part or the else part, but not both. Since we could take either one, the worst case time for the if is the slower of the two possibilities.

\[t_{\mbox{if}} = t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \]

• A missing else clause (or, for that matter, any “empty” statement list) is $O(1)$.

4.2.1 Annotating if statements

In analyzing an if statement, we will start by getting the complexity of its condition expression, its “then” part, and its “else” part.

We will record the complexity of the condition by adding an annotation of the form cond: O(...) in a comment on the line(s) where the condition is written.

Optionally, we may record the complexity of the “then” part and the “else” part in the same place as then: O(...) and else: O(...) annotations. This is optional, since, if we have clearly annotated the “then” and “else” statements, we should be able to get this information at a glance anyway.

The total complexity is recorded in the same general location using the same total: O(...) mark that we used for sequences. (In general, we use the total: mark for the total complexity of a compound statement.).

Example 3: Analyzing an If

Suppose that we have the code

if (i < j) {
 // total: O(i^2 + j)
     a(10*i+j) = foo(i,j); // O(i^2)
     bar(a(10*i+j), i, j); // O(i + j)
  }
else
   bar(0, i, j);

for which we have already analyzed the “then” part.

Before we can analyze the entire if statement, we must complete the analysis of its component pieces. First, we can do the “else” part. In the earlier example, we established that the complexity of bar was the sum of its final two parameters:

if (i < j) { 
 // total: O(i^2 + j)
     a(10*i+j) = foo(i,j); // O(i^2)
     bar(a(10*i+j), i, j); // O(i + j)
  }
else
   bar(0, i, j);  // O(i + j)

We can, if we wish, collapse the “then” part:

if (i < j) {
     // O(i^2 + j)
  }
else
   bar(0, i, j);  // O(i + j)

and then turn our attention to the condition. Relational operations on integers are $O(1)$:

if (i < j) { // cond: O(1)
     // O(i^2 + j)
  }
else
   bar(0, i, j);  // O(i + j)

We are now ready to tackle the total complexity for the if. According to our rule above,

\[ \begin{align} t_{\mbox{if}} &= t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \\ & \in O(1) + \max{(O(i^2 + j), O(i+j))} \end{align}\]

For sufficiently large values of i and j, $i^2 + j > i + j$, so

\[ \begin{align} t_{\mbox{if}} & \in O(1) + \max{O(i^2 + j), O(i+j)} \\ & = O(1) + O(i^2 + j) \\ & = O(1 + i^2 + j) \\ & = O(i^2 + j) \end{align}\]

and so

if (i < j) { // cond: O(1) total: O(i^2 + j)
     // O(i^2 + j)
  }
else
   bar(0, i, j);  // O(i + j)

If we were analyzing a still larger block of code within which this if were a single component, we might, in a separate step, collapse it, replacing it by its total complexity annotation.

4.2.2 Special Cases

The rule we have given above for the time of an if is a general rule. It’s certainly correct, but is not always going to give us the tightest bound we could get.

Sometimes we have extra knowledge about the behavior of the program that we can exploit to obtain a tighter bound. For example, if we had

i = j + 1;
   ⋮  (no changes to i or j)
if (i < j) // cond: O(1)
   {
     O(i^2 + j)
   }
else
   bar(0, i, j); // O(i + j)

we would be safe in concluding that the “then” part would never be executed, and that a better description of the behavior of this particular if would be

\[t_{\mbox{if}} = t_{\mbox{condition}} + t_{\mbox{else}} \]

in which case we would conclude that the total complexity of this if would be $O(i+j)$ instead of $O(i^2 + j)$.

Now, you might wonder if that sort of thing would ever happen in real programming. Surprisingly, it does happen.

4.3.1 Annotating Loops

Again, we can’t analyze an entire compound statement until we have analyzed its components. For loops, this means we will need to know the complexity of the body, the condition, and, for for loops, the initialization and increment portions.

We will annotate while loops like this:

while (condition)  // cond: O(...)  body: O(...) #: ...  total: O(...)
{
   body
}

• The cond: part is the complexity of the condition.

• The body: part is the complexity of the loop body, and may be omitted if it is easily read by simply looking down a couple of lines to the annotated body.

• The #: is the number of iterations in the worst case. (This is usually not a big-O expression. It can be, but most of the time we can give an exact expression for this.)

• The total: is, again, the overall complexity of the loop, combining all of its components and summing over all of the iterations.

We will annotate for loops like this:

// init: O(...)
for (init; condition; increment)    // cond: O(...)  body: O(...) incr: O(...) #:  ...  total: O(...)
{
   body
}

which adds the two for-loop specific components:

• init: is the complexity of the initialization. This is written before loop to highlight the fact that it happens only once. It can be omitted if it is $O(1)$.

• incr: part is the complexity of the increment. It can be omitted if it is $O(1)$.

\[ \begin{align*} t_{\mbox{loop}} =& t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ \in & O(1) + \sum_{j=0}^{i-1}(O(1) + O(1) + O(1))
+ O(1) \end{align*} \]

Now, just looking at the values of the times inside the summation, we can observe that none of them depend in the value of j (the particular iteration that we are in). A general special case for summations is that

If $f(\ldots)$ is independent of $i$, then $\sum_{i=1}^{n} f(\ldots) = n f(\ldots)$.

So we can immediately simplify to

\[ \begin{align*} t_{\mbox{loop}} \in& O(1) + i * (O(1) + O(1) + O(1)) + O(1) \\ =& O(1) + i*O(1) + O(1) \\ =& O(1) + O(i) + O(1) \\ =& O(i) \end{align*} \]

4.4 Recursive Functions

For recursive routines, we must take the number of recursive calls times the complexity of each call.

Our final rule for combining statements is one which we won’t use for little while, but I’ll go ahead and list it here for the sake of completeness. When we have recursive functions (functions that call themselves), things get little bit complicated. Our general approach to first find the complexity of each call while ignoring the part that calls itself, and then trying to figure out how many recursive calls are made, then add up the complexity of the individual calls across all the number of calls this function actually makes.

5 The Importance of Understanding

The rules I’ve discussed here will cover most of the circumstances you will encounter. But they aren’t a complete list of all programming language constructs and they don’t amount to a recipe that can be followed without thinking.

You need to always be aware of what the code you are analyzing actually does. If you see a statement that you just know has to manipulate K pieces of information, then any answer that isn’t O(K) or slower should be suspicious. And if you don’t know what a particular statements or group of statements actually does, you need to figure that out before you try to figure out how long it will take to do that thing.

You can't analyze the complexity of code that you don't understand.

No one can.