3 Probably Not a Problem

We’re going to need a few basic facts about probabilities for this section.

• Every probability is between $0$ and $1$, inclusive.

  • For example, the probability of a flipped coin coming up heads is $0.5$.
  • The probability of an ordinary six-sided die rolling a ‘3’ is $1/6$.
    • The probability of that same die rolling an even number is $0.5$.

• The sum of the probabilities of all possible events must be $1.0$.

  • For example, the probability of an ordinary six-sided die rolling some number between 1 and 6 is $1.0$.

• If $p$ is the probability that an event will happen, then $(1-p)$ is the probability that the event will not happen.

  • The probability of a 6-sided die rolling any number other than ‘3’ is $1 - 1/6 = 5/6$.

• If two events with probabilities $p_1$ and $p_2$ are independent (the success or failure of the first event does not affect the success or failure of the other), then

  1. the probability that both events will occur is $p_1 * p_2$.
  2. the probability that neither will occur is $(1 - p_1)(1 - p_2)$.
  3. the probability that at least one will occur is $1 - (1-p_1)(1-p_2)$.

  For example, if I roll two 6-sided dice, the probability that both will roll a ‘3’ is $\frac{1}{6} * \frac{1}{6} = \frac{1}{36}$.

  If I roll two six-sided dice, the probability that at least one will come up ‘3’ is

  \[ 1 - \left(1-\frac{1}{6}\right)\left(1-\frac{1}{6}\right) = 1 - \left(\frac{5}{6}\right)\left(\frac{5}{6}\right) = 1 - \frac{25}{36} = \frac{11}{36} \]

• More generally, if I have a series of independent events with probabilities $p_1, p_2, \ldots p_k$, then

  1. the probability that all events will occur is $\prod_{i=1}^{k} p_i$.
  2. the probability that neither will occur is $\prod_{i=1}^{k}(1 - p_i)$.
  3. the probability that at least one will occur is $1 - \prod_{i=1}^{k}(1-p_i)$.

  If I flip a coin 3 times, the probability that I will see the sequence heads-tails-heads is $\frac{1}{2} * \frac{1}{2} * \frac{1}{2} = \frac{1}{8}$

  • Note that the probability of seeing the sequence heads-heads-heads is exactly the same.
  If I flip a coin 3 times, the probability that I will see at least one heads is 1 - $\frac{1}{2} * \frac{1}{2} * \frac{1}{2} = \frac{7}{8}$

• If we make a number of independent attempts at something, and the chance of seeing an event on any single attempt is $p$, on average we will need $1/p$ attempts before seeing that event.

  • For example, if I start rolling a six-sided die until I see a ‘2’, on average I would wait $\frac{1}{1/6} = 6$ rolls.

    This is a great illustration of the difference between average and worst-case times, because in the worst case you would keep rolling forever!

4 What’s an Average?

​
For some people, average case analysis is difficult because they ​ don’t have a very flexible idea of what an “average” is.

Example:

Last semester, Professor Cord gave out the following grades in his CS361 class:

A, A, A-, B+, B, B, B, B-, C+, C, C-, D, F, F, F, F

Translating these to their numerical equivalent,

4, 4, 3.7, 3.3, 3, 3, 3, 2.7, 2.3, 2, 1.7, 1, 0, 0, 0, 0

what was the average grade in Cord’s class?

According to some classic forms of average:

Median

the middle value of the sorted list, (the midpoint between the two middle values given an even number of items)

\[ \mbox{avg}_{\mbox{median}} = 2.5 \]

Mode

the most commonly occurring value

\[ \mbox{avg}_{\mbox{modal}} = 0 \]

Mean

Computed from the sum of the elements

\[ \begin{align} \mbox{avg}_{\mbox{mean}} &= (4 + 4 + 3.7 + 3.3 + 3 + 3 + 3 + 2.7 + 2.3 + 2 + 1.7 + 1 + 0 + 0 + 0 + 0) / 16 \\ &= 2.11 \end{align} \]

​

4.1 The Mean Average

The mean average is the most commonly used, and corresponds to most people’s idea of a “normal” average, but even that comes in comes in many varieties:

Simple mean

$\bar{x} = \frac{\sum_{i=1}^N x_i}{N}$

Weighted mean

$\bar{x} = \frac{\sum_{i=1}^N w_i * x_i}{\sum_{i=1}^N w_i}$

The $w_i$ are the weights that adjust the relative importance of the scores.

Example:

Last semester Professor Cord gave the following grades

Grade	# students
4.0	2
3.7	1
3.3	1
3.0	3
2.7	1
2.3	1
2.0	1
1.7	1
1.3	0
1.0	1
0.0	4

The weighted average is $\frac{2*4.0 + 1*3.7 + 1*3.3 + 3*3.0 + 1*2.7 + 1*2.3 + 1*2.0 + 1*1.7 + 0*1.3 + 1*1.0 + 4*0.0}{2 + 1 + 1 + 3 + 1 + 1 + 1 + 1 + 0 + 1 + 4}$ $= 2.11$

Another example of weighted averages:

When one student asked about his overall grade for the semester, Professor Cord pointed out that assignments were worth 50% of the grade, the final exam was worth 30%, and the midterm exam worth 20%. The student has a B, A, and C-, respectively on these.

Category	Score	Weight
Assignments	3.0	50
Final	4.0	30
Midterm	1.7	20

So the student’s average grade was $$\frac{50*3.0 + 30*4.0 + 20*1.7}{50+30+20} = 3.04$$

​

4.2 Expected Value

The expected value is a special version of the weighted mean in which the weights are the probability of seeing each particular value.

If $x_1, x_2, \ldots ,$ are all the possible values of some quantity, and these values occur with probability $p_1, p_2, \ldots ,$, then the expected value of that quantity is

\[ E(x) = \sum_{i=1}^N p_i * x_i \]

Note that if we have listed all possible values, then

\[ \sum_{i=1}^N p_i = 1 \]

so you can regard the $E(x)$ formula above as a special case of the weighted average in which the denominator (the sum of the weights) becomes simply “1”.

Example:

After long observation, we have determined that Professor Cord tends to give grades with the following distribution:

Grade	probability
4.0	2/16
3.7	1/16
3.3	1/16
3.0	3/16
2.7	1/16
2.3	1/16
2.0	1/16
1.7	1/16
1.3	0/16
1.0	1/16
0.0	4/16

So the expected value of the grade for an average student in his class is

$$\begin{align} &((2/16)*4.0 + (1/16)*3.7 + (1/16)*3.3 + (3/16)*3.0 \\ &+ (1/16)*2.7 + (1/16)*2.3 + (1/16)*2.0 + (1/16)*1.7 + (0/16)*1.3 \\ &+ (1/16)*1.0 + (4/16)*0.0) \\ &= 2.11 \end{align}$$

The expected value is the kind of average we will use throughout this course in discussing average case complexity.

5 Determining the Average Case Complexity

In many ways, determining average case complexity is similar to determining the worst-case complexity.

5.1 Why Might Average-Case Be Smaller than Worst-Case?

It boils down to 3 possible reasons:

1. Your code calls another function whose average case complexity is smaller than its worst case.
2. You have a loop (or recursion) that, on average, does not repeat as often as it would in the worst case.
3. You have an if statement with different complexities for its then and else parts, that if statement is inside a loop (or recursion), and, on average it takes the cheaper option ore often than it would in the worst case.

5.2 It Still All Boils Down to Addition

If you want to know how much time a complicated process takes, you figure that out by adding up the times of its various components.

That basic observations is the same for average times as it is for worst-case times.

When in doubt, just add things up.

• Just keep in mind that you want to add up their average times instead of their worst-case times.

5.3 All of Your Variables Must be Defined

In math, as in programming, all of your variables must be declared/defined before you can use them.

5.4 Complexity is Written in Terms of the Inputs

The complexity of a block of code must be a function of the inputs (only!) to that block.

5.5 The Complexity of Any Block of Code Must be Numeric

No reason for this to change.

5.6 Surprises Demand Explanation

The vast majority of algorithms we will look at will have the same average-case complexity as worst-case. So, if you come up with a different value for the average-case, make sure that you understand why.

By the same token, though, if you make it to the end of an average-case analysis and never once took the time to even consider how the “average input” is different from the “worst-case input”, you may need to start over.

6 The Complexity of Expression Evaluation

Very little changes for evaluating the cost of C++ expressions:

• Arithmetic & Relational Operations on Primitive Types are O(1)

  Still true.

• Assignments of Primitive Types are O(1)

  Still true.

• Basic Address Calculations are O(1)

  Still true.

• A function calls has the average-case complexity of the function body, with appropriate substitutions of the actual parameters for the formal parameters.

  No change from worst-case analysis, except that we must have done an average-case analysis on the function body instead of a worst-case analysis.

With so much the same between average and worst-case analysis, where do the differences come into play? The differences are in the treatment of compound statements, especially loops.

7 The Complexity of Compound Statements

As in worst-case analysis, we will usually analyze compound statements in an inside-out fashion, starting with the most deeply nested components and working our way outward from there.

We will annotate our code in the same way as before to record our analysis.

7.1 Sequences of Statements

The simplest form of compound statement is the sequence or block, usually written in C++ between { } brackets. Examples include function bodies, loop bodies, and the “then” and “else” parts of if statements.

A sequence tells the machine to process statements one after the other. So,

The time for a sequence of statements is the sum of the times of the individual statements.

More formally, for a sequence of statements $s_1, s_2, …, s_k$

\[ E(t_{\mbox{seq}}) = \sum_{i=1}^k E(t_{s_i}) \]

When doing worst-case analysis, we added up the worst-case times.

When doing average-case analysis, we add up the average-case (expected value) of the times instead.

7.2 Conditional statements

When we have an if statement, we know that we will execute either the then part or the else part, but not both. However, when doing an average-case analysis we generally do not know which part we will take. Instead, we have to consider the probability of taking each part.

Let $p$ be the probability that the if condition is true (i.e., that we will take the “then” part).

\[E(t_{\mbox{if}}) = E(t_{\mbox{condition}}) + \left(p*E(t_{\mbox{then}}) + (1-p)*E(t_{\mbox{else}})\right) \]

The expression $\left(p*E(t_{\mbox{then}}) + (1-p)*E(t_{\mbox{else}})\right)$ is simply the expected value (average) of the time to take the “then” and “else” parts.

7.3 Loops

When we analyze a loop, we need to add up the expected time required for all of its iterations.

The complicating factor here is that, in worst case analysis, we can almost always say “in the worst case, for an input of size N, this loop will repeat k times”, and then sum up the times over those $k$ iterations.

When we are doing an average case analysis, we often have to consider the possibility that the number of iterations may vary, even among inputs of the same size $N$. Each possible number of iterations will have a certain probability, and we would need to get the expected value of the time over all of those possible numbers of iterations

7.3.1 Special case: constant or linear loops

Luckily, we often will not need to evaluate this formula in all of it’s generality.

For example, if $t_L(k)$ is $O(1)$ or $O(k)$ we can instead say

\[ t_{\mbox{loop}} = t_L(E(k)) \]

i.e., we simply need to find the average number of iterations and evaluate the loop time on that number.

7.3.2 Special case: a fixed number of iterations

Another common simplification occurs when we can determine the number of iterations for all inputs of size $N$. If we know that the loop executes $h(N)$ times for all inputs of size $N$, then the probabilities $p_k$ of executing the loop $k$ times is

\[ p_k = \left\{ \begin{array}{ll} 1.0 & \mbox{if } k = h(N) \\ 0.0 & \mbox{if } k \neq h(N) \\ \end{array} \right. \]

so that all of the terms in

\[ t_{\mbox{loop}} = \sum_{k=0}^{\infty} p_k t_L(k) \]

are zero except the $k = h(N)$ term:

\[ t_{\mbox{loop}} = t_L(h(N)) \]

Example 2: Filling an Array

Consider the code:

void fill_n (int* array, int n, int value)
{
   for (int i = 0; i < n; ++i)
       array[i] = value;
}

where the values in the array are known to be in the range 0..10, with half of them being zero.

What is the average case complexity of this code?

The body is O(1):

void fill_n (int* array, int n, int value)
{
   for (int i = 0; i < n; ++i)  
       array[i] = value;        // O(1)
}

The loop condition is O(1), and the loop repeats $n$ times.

void fill_n (int* array, int n, int value)
{
   for (int i = 0; i < n; ++i)  // cont: O(1)  #: n
       array[i] = value;        // O(1)
}

Now, because the number of loop iterations depends only on the size of the input ($n$) and not upon the variations in input within a given size, we can say that

\[ t_{\mbox{loop}} = t_L(n) \]

and evaluate $t_L$ according to our conventional rules:

void fill_n (int* array, int n, int value)
{
   for (int i = 0; i < n; ++i)  // cont: O(1)  #: n  total: O(n)
       array[i] = value;        // O(1)
}

and the entire function has average case complexity $O(n)$ – the same as it’s worst-case complexity. That should not be a surprise - this function behaves exactly (time-wise) the same no matter values we supply in the array or in value.

​

8 Extended Example: Ordered Insertion, Different Input Distributions

We’ll illustrate the process of doing average-case analysis by looking at a simple but useful algorithm, exploring how changes in the input distribution (the probabilities of seeing various possible inputs) affect the average case behavior.

8.1 Input in Sorted Order

template <typename Iterator, typename Comparable>
int addInOrder (Iterator start, Iterator stop,
                const Comparable& value)
{
  Iterator preStop = stop;    
  --preStop;                  
  while (stop != start && value < *preStop) {
    *stop = *preStop;         
    --stop;                   
    --preStop;                
  }
  // Insert the new value
  *stop = value;              
  return stop;                
}

If the input arrives in sorted order, then each call to the function will execute the loop zero times, because the word being inserted will always be alphabetically greater than all the words already in the container.

So if $p_k$ denotes the probability of executing the loop $k$ times, then \( p_0=1, p_1 = 0, p_2 = 0, \ldots \) .

So the time is

\[ \begin{align} t_{\mbox{loop}} & = t_L(0) \\ & = O(1) \end{align} \]

For this input pattern, the entire algorithm has an average-case complexity of $O(1)$.

8.2 Input in Arbitrary Order:

template <typename Iterator, typename Comparable>
int addInOrder (Iterator start, Iterator stop,
                const Comparable& value)
{
  Iterator preStop = stop;    
  --preStop;                  
  while (stop != start && value < *preStop) {
    *stop = *preStop;         
    --stop;                   
    --preStop;                
  }
  // Insert the new value
  *stop = value;              
  return stop;                
}

In this case, we are equally likely to need 0 iterations, 1 iteration, 2 iterations , … , n iterations, where $n$ is distance(start,stop). So the possible numbers of iterations from 0 to $n$ are all equally likely:

\[p_i = \left\{ \begin{array}{ll}\frac{1}{n+1} & \mbox{if } 0 \leq k \leq n \\ 0 & \mbox{otherwise}\end{array}\right. \]

The cost of the loop condition and of the body is constant for each iteration, however, so we can use the special case

\[ t_{\mbox{loop}} = t_L(E(k)) \]

where $E(k)$ is the expected number of iterations of the loop.

What is $E(k)$?

Intuitively, if we are equally likely to repeat the loop 0 times, 1 time, 2 times, … , $n$ times, the average number of iterations would seem to be $n/2$.

Formally,

\[ \begin{eqnarray*} E(k) & = & \sum_{k=0}^{\infty} p_k k \\ & = & \sum_{k=0}^{\mbox{n}} p_k k \; \; (\mbox{because } p_k=0 \mbox{ when } k > \mbox{n})\\ & = & \sum_{k=0}^{\mbox{n}} \frac{1}{\mbox{n}+1} k \\ & = & \frac{1}{\mbox{n}+1} \sum_{k=0}^{\mbox{n}} k \\ & = & \frac{1}{\mbox{n}+1} \frac{\mbox{n}(\mbox{n}+1)}{2} \\ & = & \frac{\mbox{n}}{2} \\ \end{eqnarray*} \]

Chalk one up for intuition!

So the loop is $\frac{n}{2} O(1) = O(n)$

template <typename Iterator, typename Comparable>
int addInOrder (Iterator start, Iterator stop, const Comparable& value)
{
  Iterator preStop = stop;    // O(1)
  --preStop;                  // O(1)
  while (stop != start && value < *preStop) { //cond: O(1) #:  n/2  total: O(n)
    // O(1)
  }
  // Insert the new value
  *stop = value;              // O(1)
  return stop;                // O(1)
}

And we can then replace the entire loop by $O(\mbox{n})$.

template <typename Iterator, typename Comparable>
int addInOrder (Iterator start, Iterator stop, const Comparable& value)
{
  Iterator preStop = stop;    // O(1)
  --preStop;                  // O(1)
  // O(n)
  // Insert the new value
  *stop = value;              // O(1)
  return stop;                // O(1)
}

And now, we add up the complexities in the remaining straight-line sequence, and conclude that the entire algorithm has an average case complexity of $O(n)$, where $n$ is the distance from start to stop, when presented with randomly arranged inputs.

This is the same result we had for the worst case analysis. Does this mean that it runs in the same time on average as it does in the worst case? No, on average, it runs in half the time of its worst case, but that’s only a constant multiplier, so it disappears when we simplify.

Under similar randomly arranged inputs, the average case complexity of ordered search is $O(n)$ and the average case complexity of binary search is $O(\log n)$. Again, these are the same as their worst-case complexities.

​

8.3 Inputs in Almost-Sorted Order

​
We’ve already considered the case where the inputs to this function were already arranged into ascending order. What would happen if the inputs were almost, but not exactly, already sorted into ascending order?

For example, suppose that, on average, one out of $n$ items is out of order. Then the probability of a given input repeating the loop zero times would be $p_{\mbox{0}} = \frac{n-1}{n}$, and some single $p_{\mbox{i}}$ would have probability $1/n$, with all the other probabilities being zero.

Assuming the worst (because we want to find an upper bound), let’s assume that the one out-of-order element is the very last one added, and that it actually gets inserted into position 0. Then we have $p_0 = (n-1)/n, p_1 = 0, p_2 = 0, … , p_{n-1} = 0, p_n = 1/n$

So the average number of iterations would be given by \begin{eqnarray*} E(k) & = & \sum_{k=0}^{n} k p_k \\ & = & 0 * (n-1)/n + n * 1/n \\ & = & n/n \\ & = & 1 \\ \end{eqnarray*} and the function is $O(E(k)) = O(1)$

8.4 Almost Sorted - version 2

Now, that’s only one possible scenario in which the inputs are almost sorted. Let’s look at another. Suppose that we knew that, for each successive input, the probability of it appearing in the input $m$ steps out of its correct position is proportional to $1/(m+1)$ (i.e., each additional step out of its correct position is progressively more unlikely). Then we have $p_{0}=c, p_{1}=c/2, p_{2}=c/3, … p_{n-1}=c/n, p_{n}=c/(n+1)$.

The constant $c$ is necessary because the sum of all the probabilities must be exactly 1. We can compute the value of $c$ by using that fact:

\begin{align} \sum_{i=0}^{n} p_i = & 1 \\ \sum_{i=0}^{n} \frac{c}{i+1} = & 1 \\ c \sum_{i=0}^{n} \frac{1}{i+1} = & 1 \end{align}

This sum, for reasonably large n, is approximately $\log n$.

So we conclude that $c$ is approximately $= 1/\log(n)$.

So the function, for this input distribution, is

\begin{align} t_{\mbox{loop}} = & O(E(k)) \\ = & O\left(\sum_{i=0}^n (i + 1)p_i\right) \\ = & O\left(\sum_{i=0}^n (i + 1) \frac{c}{i+1}\right) \\ = & O\left(\sum_{i=0}^n c\right) \\ = & O((n+1)c) \\ = & O\left(\frac{n}{\log n}\right) \end{align}

So the average case is slightly smaller than the worst case, though not by much (remember that $\log n$ is nearly constant over large ranges of $n$, so $n/(\log(n))$ grows only slightly slower than $n$.

9 The Input Distribution is Key

You can see, then, that average case complexity can vary considerably depending upon just what constitutes an “average” set of inputs.

Utility functions that get used in many different programs may see different input distributions in each program, and so their average performances in different programs will vary accordingly.