Worst-Case Complexity Analysis
Steven J. Zeil
Now that we have seen how to manipulate big-O expressions, the next step is to figure out where we get them from in the first place. We are going to consider common programming language constructs and how we can determine the worst-case behavior for each of them, starting with simple expressions and building up to entire function bodies.
As we do this, we will be looking at two things:
-
How do we derive the worst-case expression for each kind of programming language construct?
-
How can we annotate our code to express those worst-case behaviors in a way that amounts to a proof of your results?
A worst-case analysis is really a kind of mathematical proof, and people’s styles of writing mathematical proofs vary so much that the results are often unreadable by anyone other than the author.
So in this lesson I am going to show you my own copy-and-paste style for conducting and documenting a worst-case analysis. This is the technique that I will use through this course whenever I am giving an analysis. And it is also the technique that I will expect you to use when you do assignments in which I ask you to show me your work on the algorithm analysis.
This technique takes advantage of the fact that these days we don’t usually write down our algorithms using paper and pencil. Instead we use text editors which are very good in doing copy and paste. Consequently the technique relies on your ability to make copies of an algorithm over and over again, making small changes to the algorithm at each stage that reflect the analysis that you have actually done.
1 Some Overall Guidelines
First, some basic observations.
1.1 It All Boils Down to Addition
If you want to know how much time a complicated process takes, you figure that out by adding up the times of its various components.
Now, we will have a lot of situations where we use multiplication and other operations, but those are always “special cases”. For example, if I tell you that a certain component of a program takes time $t_0$ and we later discover that this component is inside a loop that repeats 1000 times, we may write the total as $1000 t_0$, but that’s only because
\[ 1000 t_0 = \sum_{i=1}^{1000} t_0 = t_0 + t_0 + \ldots (997 \mbox{ times})\ldots + t_0 \]
It’s just a simplification for a lot of additions.
When in doubt, just add things up.
1.2 Nothing Happens in O(0) Time
Even doing nothing takes time. “Not doing” a block of code still requires at least a clock cycle to move the address counter forward to the next block of code. So, even doing nothing is $O(1)$, not $O(0)$.
So if your math comes out to zero (or even worse, negative time values), it’s time to go back and check your math.
1.3 All of Your Variables Must be Defined
In math, as in programming, all of your variables must be declared/defined before you can use them.
If you reach a conclusion that a certain C++ function is $O(N + M^2)$, then $N$ and $M$ have to actually mean something. That meaning may come because they are actual variables in the code:
void foo(double[] array, int N, int M) // O(N + M^2)
or because, in the style of mathematical proofs everywhere, we have supplied our own definition, e.g.,
void foo (vector<double> v)
Let N be the number of elements in the vector v and let M
be the number of those elements that are greater than 1.0.
But, one way or another, every symbol in your big-O expressions needs to be properly declared.
- Don’t fall in love with “n”. Even though our definition of complexity talks about $\bar{n}$, that doesn’t mean that there has to be a variable named “n” in the code, or that “n” means anything at all in a specific analysis problem.
- And, don’t forget that , in mathematics as in C++,
n
andN
are not the same variable.
The possibility that we may use variables from the code in our mathematical description of complexity leads to a closely related rule:
1.4 Complexity is Written in Terms of the Inputs
The complexity of a block of code must be a function of the inputs (only!) to that block.
If you are looking at a block of code like this:
int x = y + z;
int k = foo (x, y, z);
bar (k);
your big-O expression to describe this code cannot involve $k$. That’s because $k$ does not even exist when we start the block of code. And, in many cases, e.g.:
void baz(int y, int z)
{
int x = y + z;
int k = foo (x, y, z);
bar (k);
}
$k$ will not exist after we have completed the block of code. In this case, because $k$ is computed from x
, y
, and z
, and x
was computed from y
and z
, we would need to say, instead of $O(k)$, the complexity could be $O(\mbox{foo}(y+z,y,z))$.
1.5 The Complexity of Any Block of Code Must be Numeric
Our definition of $t \in O(f(\bar{n}))$ says that “… $t < c*f(\bar{n})$…”.
The “<” and multiplication by a constant $c$ only makes sense if $f(\bar{n})$ is a numeric quantity of some kind. So if we were looking at
void foo (vector<string> v, string str)
it might be valid to say that the complexity of foo
is O(1)
, O(v.size())
, O(N)
where N is the number of vowels in str
, etc., because all of those are numeric quantities. But you cannot possibly say the complexity is O(v)
or O(str)
because v
and str
are not numeric. They can’t be multiplied by a constant c
and cannot be compared to a numeric time value.
1.6 Surprises Demand Explanation
In many scientific and engineering fields, students are taught that any complicated calculations should be subjected to “sanity checks” that may
-
check parts of the calculation for internal consistency
- Many of the rules we’ve already been discussing in this section would fall into this category
-
compare the answer against a simpler, rough calculation (often called a “back-of-the-envelope” method, after the stereotypical image of an engineer scratching a few numbers on a handy piece of paper and suddenly announcing an approximate answer to the problem that everyone else has been struggling with).
Back-of-the-envelope calculations are not a substitute for doing the “real” problem solution, but they do provide an important function as a sanity check. It’s entirely possible that, when we do the real calculation, we will come up with a different answer than the back-of-the-envelope answer. If so, it’s worthwhile to take a moment to understand why the two answers are different.
- Maybe we made a mistake in one computation or the other. If so, we need to find that mistake and correct it.
- On the other hand, maybe we did everything correctly, and there is some deep feature of this problem that isn’t obvious to a quick back-of-the-envelope approach, something that accounts for the surprise of the final result of the full calculation.
If so, we need to identify and explain that surprise before we can have full confidence in our final answer.
Example 1: A back-of-the-Envelope method for loopsFor example, most of the time of typical code is spent in loops, and so loops tend to dominate the worst-case complexity. A reasonable back-of-the-envelope technique is to
- Look at the most deeply nested loops in the algorithm. If the loops are nested $k$ deep, and each appears to execute up to $N$ times, then $N^k$ is a reasonable first guess for the algorithm complexity.
It’s only a guess, but if you come up with something else, you should be sure that you understand where the difference is coming from.
2 The Complexity of Expression Evaluation
Most of the actual executable part of a typical C++ program is made up of expression evaluation. Even many things that we refer to as statements, such as assignment statements or procedure/function calls, e.g.
x = y + z;
doSomethingWith(x,y,z);
are just larger expressions in C++.
2.1 Arithmetic & Relational Operations on Primitive Types are O(1)
If we restrict our attention to the primitive types (integers, floating point numbers, booleans, characters, pointers, and references, then all operators provided by the language are O(1).
Think about something like:
int x = ...
int y = ...
⋮
x + y // What is the complexity of this?
the time required to compute the sum is independent of how large the values in x
and y
might be. It takes no more time to compute 1000000 + 987654321
that it takes to compute 0 + 1
. It might take a bit longer to add a pair of 64-bit long
integers than a pair of 8-bit short
integers, but the available bit widths is fixed by the environment (CPU and compiler) and, for each available choice of widths, is a constant no matter what the actual numbers involved might be.
The same is true for subtraction, multiplication, boolean operations, relational operators, etc.
When we combine these operators into more complicated expressions, we are simply instructing the machine to do one of these operations, then another, and so on. For example, if we had integer variable x, y, and z, then an expression like x + y*z
is $O(1)$. There are two operations in that expression, a multiplication and an addition, and those operations each run in $O(1)$ time (i.e., they each take a constant amount of time regardless of the amount of data being manipulated by the program). So the expression has complexity $O(c_+ + c_*)$, which, by our rules for big-O algebra, simplifies to $O(1)$.
On the other hand the complexity of an expression like foo(i) +
bar(j)
would be the sum of the complexities of the calls to foo and to bar. We’ll talk about how those calls are evaluated shortly.
Question: What is the worst-case complexity of the expression `(x < 100) & (y + 1 > 0)``?
2.2 Assignments of Primitive Types are O(1)
Really, assignments of any data type that has a fixed number of bytes (ints, doubles, chars, etc., but not strings, vectors, etc.) are O(1).
The explanation is that it takes the same amount of time to copy a byte that contains 127 as a byte that contains 0. The actual value stored there is irrelevant. The number of bytes making up the total does matter, but if that is fixed, then copying that number of bytes takes the same amount of time for all data types of that size.
- Again, this does not mean that assignment of non-primitive types is always O(1). C++ allows programmers to write their own assignment operators, and until we actually examine the algorithm used for such an operator, we can’t guess at what its complexity would be.
2.3 Basic Address Calculations are O(1)
We have previously discussed the fact that array indexing is just an address calculation and shown that it boils down to an integer multiplication and addition. So
- Array indexing
a[i]
is $O(1)$.-
That doesn’t mean that everything we do with array elements will be $O(1)$. For example, if I write
int array[100]; ⋮ a[i] = a[i-1]; // O(1)
the whole expression, including the assignment, is composed of $O(1)$ components, so the whole thing is $O(1)$.
But if I had
string array[100]; ⋮ a[i] = a[i-1]; // _Not_ O(1)
the final statement is probably not $O(1)$. But that’s because assignment of strings
s = t
is generallyO(t.length())
. It has nothing to do with the time required to compute the addressesa[i]
anda[i-1]
.
-
Similarly,
-
Pointer dereferencing
*ptr
is O(1). This operation simply grabs an address out of memory and waits to see what we want to do with it.Again, this does not mean that the uses we make of those addresses will be $O(1)$. If I write, for example,
*ptr1 = *ptr2
, I really need to focus on the complexity of assignment for whatever data type those pointers point to. -
Struct/class member selection
myStruct.member
is $O(1)$. The ‘.’ is just an address calculation in which the base address of the entiremyStruct
structure is added together with a compiler-assigned integer offset for the beginning ofmember
within each structure of that same type. It’s just an addition. -
The combined pointer dereference and member select
ptr->member
is $O(1)$.It’s just the combination of two $O(1)$ components.
2.4 Function Calls
So far, everything we might write into an expression has been pretty quick. That stops once we start to consider function calls (including programmer-supplied operator
overloads).
Addition is still king. When we have an expression like
int k = foo (x, y, z);
we still add the complexity of the function call to the complexity of the other operations (in this case, integer assignment). But what is the complexity of the function call?
To answer that, someone needs to have analyzed the body of that function and figured out its complexity.
-
Sometimes someone else will have already done that for us. In particular, many functions in the
std
library are required by the C++ standard to have a certain worst-case complexity. So in that case we can simply look it up. -
If no one else has done the job for us, we will have to set our current problem aside for the moment and analyze the function body ourselves. A fair number of functions form the
std
library are so simple that, by a few weeks from now, you’ll be able to analyze them at a glance. Other functions may require a substantial effort.
For now, let’s assume that, one way or the other, we have found the complexity of a function that we are calling. What does that mean? What will we actually “know”?
Remember that “complexity is written in terms of the inputs”. So what we will get is a big-O expression written in terms of the possible inputs to the function.
For example, the worst-case complexity of the assignment operator for strings, const string& operator= (const string& t)
, is O(t.length())
.
- Take note that
operator=
is a member function of classstring
. Like all member functions, it has a hidden parameter named “this”. So, in general, “this
” is an input to member functions and we might expect a member function’s complexity to include either explicit or implicit references tothis->
.
Now the input variables that are listed in the complexity of a function will be the formal parameters of the function. We will need to replace those by the actual parameters that we actually use in the function call.
For example, earlier we looked at
string array[100];
⋮
a[i] = a[i-1]; // _Not_ O(1)
What is the complexity of the expression in the last line of code there?
- We have said that the complexity of the address calculations
a[i]
anda[i-1]
are $O(1)$. - We have said that the complexity of the function
const string& operator= (const string& t)
, isO(t.length())
.- But
t
is a formal parameter, for which this particular line of code substitutesa[i-1]
. - Therefore the complexity of that function call is
O(a[i-1].length())
.
- But
The total complexity of that line of code, therefore, is O(1) +
O(1) + O(a[i-1].length())
, which simplifies to O(a[i-1].length())
.
Question: Let $L$ denote the length of the longest string in array
. Would it then be valid to state that the above line of code is $O(L)$?
2.5 Annotating Expression Statements
As promised at the beginning of this lesson, we will be recording our conclusions about the complexity of the various components of the code by annotating the code itself with comments.
For statement-level expressions (e.g., assignments or void
function calls) the annotation is very simple. We simply write the big-O expression at the end of the line as a //
comment. In fact, you have already seen me do this:
int array[100];
⋮
a[i] = a[i-1]; // O(1)
and
string array[100];
⋮
for (int i = N; i > 0, --i)
a[i] = a[i-1]; // O(a[i-i].length())
Now, typing mathematics in plain text like this has some limitations that we will need to work around.
-
The most common is the need to use superscripts to denote raising something to a power. We will use the caret (
^
) character for that purpose, for example, typing $O(N^2)$ asO(N^2)
.Not only does
^
suggest raising something up, but in some programming languages, it is actually used as the “raise to a power” operator. -
Less often, we will want to denote subscripts. We will use the underscore (
_
) character for that purpose. For example, we would type $O(x_0 + x_1)$ asO(x_0 + x_1)
.
As we conduct an analysis of complexity, we will wind up with these kinds of annotations sprinkled throughout the code, e.g.,
template <typename Comparable>
int seqOrderedSearch(const Comparable list[], int listLength,
Comparable key)
{
int loc = 0; // O(1)
while (loc < listLength && list[loc] < key)
{
++loc; // O(1)
}
return loc; // O(1)
}
(A return statement is essentially a copy or assignment to a special variable in the caller, so for primitive types it, like assignment, will be $O(1)$.)
3 The Complexity of Compound Statements
Of course, expression evaluation and assignment may account for the bulk of single statements in a program, but what allows us to combine single statements into a full algorithm is the idea of compound statements: sequences (a.k.a. blocks), conditionals (e.g., if
), and loops.
Compound statements are single statements that are built around one or more simpler component statements. For example, a while loop is build around a loop body, which is itself a statement. Some of the components of a compound statement may also be smaller compound statements, as when we nest loops inside one another.
There is a general observation we can make about analyzing compound statements. In order to compute an upper bound on the time required by a compound statement, we will almost always need to know the bounds on the times of its components. Consequently
We usually analyze compound statements in an inside-out fashion, starting with the most deeply nested components and working our way outward from there.
3.1 Sequences of Statements
The simplest form of compound statement is the sequence or block, usually written in C++ between { }
brackets. Examples include function bodies, loop bodies, and the “then” and “else” parts of if
statements.
A sequence tells the machine to process statements one after the other. So,
The time for a sequence of statements is the sum of the times of the individual statements.
More formally,for a sequence of statements $s_1, s_2, …, s_k$
\[ t_{\mbox{seq}} = \sum_{i=1}^k t_{s_i} \]
This rule applies when we have a block of statements in a straight line – no way to jump into the middle of the block, no way to jump out. When we have sequential arrangements of statements like that, then the complexity of the sequence as a whole is sum of the complexities of the statements
Example 2: Statement Sequence{ a[10*i+j] = foo(i,j); bar(a[10*i+j], i, j); }
In the code shown here, if I look at the first assignment statement, its complexity depends on the complexity of the body of the function
foo
.Let’s just say for the sake of example that the function
foo(x,y)
has a body of complexity $O(x^{2})$. Substituting the actual parameteri
from the call for the formal parameterx
, we conclude that the first statement has complexity $O(i^{2})$We annotate the first statement accordingly:
{ a[10*i+j] = foo(i,j); // O(i^2) bar(a[10*i+j], i, j); }
The second statement involves a call to
bar
. For the sake of example, let’s assume thatbar(x,y,z)
has complexity of $O(y+z)$. Again, to get the complexity of the call we must substitute the actual parameters for the formals, and doing so indicates that the second statement has complexity $O(i + j)$.We would record our conclusion by annotating that statement:
{ a[10*i+j] = foo(i,j); // O(i^2) bar(a[10*i+j], i, j); // O(i + j) }
That would mean that the complexity for the sequence of statements consisting these two individual statements would be sum of those two complexities which would be $O(i^{2}+i+j)$ and since $i^{2}$ is greater than $i$, this simplifies to $O(i^{2} + j)$.
3.1.1 Annotating Statement Sequences
In some cases we won’t bother annotating the total complexity of a statement sequence, particularly when it is short or the answer is obvious (e.g., because all of the component statements have the same complexity).
When we do want to record the total for a statement sequence, we will do so by adding an annotation // total: O(...)
at the top of the sequence.
In our example above, we would record our conclusion like this:
{ // total: O(i^2 + j)
a(10*i+j) = foo(i,j); // O(i^2)
bar(a(10*i+j), i, j); // O(i + j)
}
As we proceed through a proof, we may decide that we don’t need the details inside the sequence any more. In that case, we may opt to make a separate copy of the code, replace the entire sequence with its total complexity, and add a brief text in between explaining what we are doing, e.g.,
{ // total: O(i^2 + j)
a(10*i+j) = foo(i,j); // O(i^2)
bar(a(10*i+j), i, j); // O(i + j)
}
Collapsing that block,
{
// O(i^2 + j)
}
3.2 Conditional statements
When we have an if
statement, we know that we will execute either the then
part or the else
part, but not both. Since we could take either one, the worst case time for the if
is the slower of the two possibilities.
\[t_{\mbox{if}} = t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \]
- A missing
else
clause (or, for that matter, any “empty” statement list) is $O(1)$.
3.2.1 Annotating if statements
In analyzing an if
statement, we will start by getting the complexity of its condition expression, its “then” part, and its “else” part.
We will record the complexity of the condition by adding an annotation of the form cond: O(...)
in a comment on the line(s) where the condition is written.
Optionally, we may record the complexity of the “then” part and the “else” part in the same place as then: O(...)
and else: O(...)
annotations. This is optional, since, if we have clearly annotated the “then” and “else” statements, we should be able to get this information at a glance anyway.
The total complexity is recorded in the same general location using the same total: O(...)
mark that we used for sequences. (In general, we use the total:
mark for the total complexity of a compound statement.).
Example 3: Analyzing an IfSuppose that we have the code
if (i < j) { // total: O(i^2 + j) a(10*i+j) = foo(i,j); // O(i^2) bar(a(10*i+j), i, j); // O(i + j) } else bar(0, i, j);
for which we have already analyzed the “then” part.
Before we can analyze the entire
if
statement, we must complete the analysis of its component pieces. First, we can do the “else” part. In the earlier example, we established that the complexity ofbar
was the sum of its final two parameters:if (i < j) { // total: O(i^2 + j) a(10*i+j) = foo(i,j); // O(i^2) bar(a(10*i+j), i, j); // O(i + j) } else bar(0, i, j); // O(i + j)
We can, if we wish, collapse the “then” part:
if (i < j) { // O(i^2 + j) } else bar(0, i, j); // O(i + j)
and then turn our attention to the condition. Relational operations on integers are $O(1)$:
if (i < j) // cond: O(1) { // O(i^2 + j) } else bar(0, i, j); // O(i + j)
We are now ready to tackle the total complexity for the
if
. According to our rule above,\[ \begin{align} t_{\mbox{if}} &= t_{\mbox{condition}} + \max(t_{\mbox{then}}, t_{\mbox{else}}) \\ & \in O(1) + \max{(O(i^2 + j), O(i+j))} \end{align}\]
For sufficiently large values of
i
andj
, $i^2 + j > i + j$, so\[ \begin{align} t_{\mbox{if}} & \in O(1) + \max{O(i^2 + j), O(i+j)} \\ & = O(1) + O(i^2 + j) \\ & = O(1 + i^2 + j) \\ & = O(i^2 + j) \end{align}\]
and so
if (i < j) // cond: O(1) total: O(i^2 + j) { // O(i^2 + j) } else bar(0, i, j); // O(i + j)
If we were analyzing a still larger block of code within which this if
were a single component, we might, in a separate step, collapse it, replacing it by its total complexity annotation.
3.2.2 Special Cases
The rule we have given above for the time of an if
is a general rule. It’s certainly correct, but is not always going to give us the tightest bound we could get.
Sometimes we have extra knowledge about the behavior of the program that we can exploit to obtain a tighter bound. For example, if we had
i = j + 1;
⋮ (no changes to i or j)
if (i < j) // cond: O(1)
{
O(i^2 + j)
}
else
bar(0, i, j); // O(i + j)
we would be safe in concluding that the “then” part would never be executed, and that a better description of the behavior of this particular if
would be
\[t_{\mbox{if}} = t_{\mbox{condition}} + t_{\mbox{else}} \]
in which case we would conclude that the total complexity of this if
would be $O(i+j)$ instead of $O(i^2 + j)$.
Now, you might wonder if that sort of thing would ever happen in real programming. Surprisingly, it does happen.
-
Programmers sometimes engage in defensive programming, writing conditions into code to defend against possible coding errors.
-
Coding standards, particularly when enforced by automatic tools, can force programmers to insert code that they believe will never be executed, just to get the standards enforcer off their backs. For example, an automated standards tool might flag a line of code
a[i] = 0;
as a “possible out-of-range error”. To get rid of that message, the programmer might write
if ((i >= 0) && (i < N)) a[i] = 0; else exit_with_message("This should never happen.");
-
When we put if statements inside loops, we will sometimes have enough knowledge of how the algorithm works to say things like “if this loop executes N times, then on $\sqrt{N}$ iterations the
if
inside will take the ‘else’ branch, but on $N - \sqrt{N}$ iterations it will take the ‘then’ branch.”If we have that kind of knowledge, we are allowed, indeed, urged to exploit it when doing so give us a tighter bound.
Complexity analysis is not a recipe to be followed exactly the same way every time we apply it. It is, like programming itself, a creative mathematical process that calls for us to use the knowledge at hand to get the best result we can.
We have general rules for compound statements that will give us a valid upper bound when applied “inside-out” from the most deeply nested statements. But sometimes we can get tighter bounds by applying our knowledge and understanding of the code to identify special cases.
3.3 Loops
When we analyze a loop, we need to add up the time required for all of its iterations.
When we encounter a loop of the form
while (condition)
{
body
}
we can say that
\[ t_{\mbox{while}} = \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \]
where
- $t_{\mbox{condition}}$ is the time to evaluate the loop condition,
- $t_{\mbox{condition}}^{\mbox{final}}$ is the time to evaluate the loop condition the final time (when we exit the loop), and
- $t_{\mbox{body}}$ is the time required to do the loop body.
The summation is taken over all iterations of the loop. The number of iterations is likely to depend on the input size $\bar{n}$ and, in many cases, the particular set of inputs that constitutes the worst case input over all possible inputs of that size. Similarly, the values of $t_{\mbox{condition}}$, $t_{\mbox{condition}}^{\mbox{final}}$, and $t_{\mbox{body}}$ may depend on the size of the input, on the particular values of the worst case input, and on which iteration we are performing.
If we encounter a for
loop:
for (init; condition; increment)
{
body
}
we know that this is equivalent to
init;
while (condition)
{
body
increment
}
and so we can substitute into our while-loop rule to get
\[ t_{\mbox{for}} = t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \]
where
- $t_{\mbox{init}}$ is the time required to do the loop initialization,
- $t_{\mbox{condition}}$ is the time to evaluate the loop condition,
- $t_{\mbox{condition}}^{\mbox{final}}$ is the time to evaluate the loop condition the final time (when we exit the loop), and
- $t_{\mbox{body}}$ is the time required to do the loop body.
3.3.1 Annotating Loops
Again, we can’t analyze an entire compound statement until we have analyzed its components. For loops, this means we will need to know the complexity of the body, the condition, and, for for
loops, the initialization and increment portions.
We will annotate while loops like this:
while (condition) // cond: O(...) body: O(...) #: ... total: O(...)
{
body
}
-
The
cond:
part is the complexity of the condition. -
The
body:
part is the complexity of the loop body, and may be omitted if it is easily read by simply looking down a couple of lines to the annotated body. -
The
#:
is the number of iterations in the worst case. (This is usually not a big-O expression. It can be, but most of the time we can give an exact expression for this.) -
The
total:
is, again, the overall complexity of the loop, combining all of its components and summing over all of the iterations.
We will annotate for loops like this:
// init: O(...)
for (init; condition; increment) // cond: O(...) body: O(...) incr: O(...) #: ... total: O(...)
{
body
}
which adds the two for-loop specific components:
-
init:
is the complexity of the initialization. This is written before loop to highlight the fact that it happens only once. It can be omitted if it is $O(1)$. -
incr:
part is the complexity of the increment. It can be omitted if it is $O(1)$.
3.3.2 Examples
Example 4: A simple loopfor (int j = 0; j < i; ++j) a[i+10*j] = i + j;
For the loop shown here, we can see that the complexity of the loop body is $O(1)$ (by our assignment statement rule).
for (int j = 0; j < i; ++j) a[i+10*j] = i + j; // O(1)
Similarly, the time to do the loop initialization, to evaluate the loop condition (
j < i
), and to increment j are also $O(1)$.// init: O(1) for (int j = 0; j < i; ++j) // cond: O(1) incr: O(1) a[i+10*j] = i + j; // O(1)
How many times does this loop execute in the worst case? Well, as is often the case with
for
loops, that’s actually pretty easy to read out of the loop header:// init: O(1) for (int j = 0; j < i; ++j) // cond: O(1) incr: O(1) #: i a[i+10*j] = i + j; // O(1)
And now we are in a position to apply our rule for for-loops:
\[ \begin{align*} t_{\mbox{loop}} =& O\left(t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}}\right) \\ \in & O(1) + \sum_{j=0}^{i-1}(O(1) + O(1) + O(1))
+ O(1) \\ = & O(1) + \sum_{j=0}^{i-1} O(1 +1 + 1) + O(1) \\ = & O(1) + \sum_{j=0}^{i-1} O(1) + O(1) \\ = & O(1) + O\left(\sum_{j=0}^{i-1} 1 \right) + O(1) \\ = & O(1) + O(i) + O(1) \\ = & O\left(1 + i + 1)\right) \\ = & O(i) \end{align*} \]And we write our conclusion that
// init: O(1) for (int j = 0; j < i; ++j) // cond: O(1) incr: O(1) #: i total: O(i) a[i+10*j] = i + j; // O(1)
We could have reached the same conclusion by a slightly different path:
\[ \begin{align*} t_{\mbox{loop}} =& t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ \in & O(1) + \sum_{j=0}^{i-1}(O(1) + O(1) + O(1))
+ O(1) \end{align*} \]
Now, just looking at the values of the times inside the summation, we can observe that none of them depend in the value of j
(the particular iteration that we are in). A general special case for summations is that
If $f(\ldots)$ is independent of $i$, then $\sum_{i=1}^{n} f(\ldots) = n f(\ldots)$.
So we can immediately simplify to
\[ \begin{align*} t_{\mbox{loop}} \in& O(1) + i * (O(1) + O(1) + O(1)) + O(1) \\ =& O(1) + i*O(1) + O(1) \\ =& O(1) + O(i) + O(1) \\ =& O(i) \end{align*} \]
Example 5: Nested LoopsSuppose we have:
for (int i = 0; i < n; ++i) for (int j = 0; j < i; ++j) a[i+10*j] = i + j;
We have already analyzed the inner loop:
for (int i = 0; i < n; ++i) { // init: O(1) for (int j = 0; j < i; ++j) // cond: O(1) incr: O(1) #: i total: O(i) a[i+10*j] = i + j; // O(1) }
and this might be a good time to collapse the inner loop:
for (int i = 0; i < n; ++i) { // O(i) }
The remaining outer loop has components
// init: O(1) for (int i = 0; i < n; ++i) // cond: O(1) incr: O(1) #: n { // O(i) }
So now we can analyze the outer loop as
\[ \begin{align*} t_{\mbox{loop}} =& t_{\mbox{init}} + \sum_{\mbox{iterations}}(t_{\mbox{condition}} + t_{\mbox{increment}} + t_{\mbox{body}}) + t_{\mbox{condition}}^{\mbox{final}} \\ \in& O(1) + \sum_{i=0}^{n-1} (O(1) + O(i) + O(1)) + O(1) \\ =& O(1) + \sum_{i=0}^{n-1} O(i) + O(1) \end{align*} \]
We can’t apply the same replace-summation-by-multiplication shortcut this time, because the expression inside the sum depends on the iteration number (
i
).So, continuing on,
\[ \begin{align*} t_{\mbox{loop}} \in& O(1) + O\left(\sum_{i=0}^{n-1} i\right) \\ \end{align*} \]
Now, that summation is actually very well known, and one that we will encounter a lot during this course. You can find it and several others in the FAQ
\[ \begin{align*} t_{\mbox{loop}} \in& O(1) + O\left(\sum_{i=0}^{n-1} i\right) \\ =& O(1) + O\left(\frac{(n-1)n}{2}\right) \\ =& O(1) + O\left((n-1)n\right) \\ =& O\left(n^2 - n + 1\right) \\ =& O\left(n^2\right) \\ \end{align*} \]
And so we conclude that
// init: O(1) for (int i = 0; i < n; ++i) // cond: O(1) incr: O(1) #: n total: O(n^2) { // O(i) }
3.4 Recursive Functions
For recursive routines, we must take the number of recursive calls times the complexity of each call.
Our final rule for combining statements is one which we won’t use for little while, but I’ll go ahead and list it here for the sake of completeness. When we have recursive functions (functions that call themselves), things get little bit complicated. Our general approach to first find the complexity of each call while ignoring the part that calls itself, and then trying to figure out how many recursive calls are made, then add up the complexity of the individual calls across all the number of calls this function actually makes.
4 The Importance of Understanding
The rules I’ve discussed here will cover most of the circumstances you will encounter. But they aren’t a complete list of all programming language constructs and they don’t amount to a recipe that can be followed without thinking.
You need to always be aware of what the code you are analyzing actually does. If you see a statement that you just know has to manipulate K pieces of information, then any answer that isn’t O(K) or slower should be suspicious. And if you don’t know what a particular statements or group of statements actually does, you need to figure that out before you try to figure out how long it will take to do that thing.
You can't analyze the complexity of code that you don't understand.
No one can.
For example, no one has been able to prove if this code
unsigned collatz (unsigned n)
{
while (n != 1)
{
if (n % 2 == 0) // n is even
n = n/2;
else // n is odd
n = 3*n + 1;
}
return 1;
}
actually comes to a halt for all values of n
. It’s been checked for all values of n
up to some absurdly high value, and has eventually halted each time, but no one has been able prove that it halts for all inputs. So it’s entirely possible that this code has worst-case complexity $O(\infty)$. But we don’t know that is true, because no one actually understands the behavior of this function well enough to answer the question of how many repetitions it does for arbitrary values of $n$.