1 The Algorithm

This is the insertion sort:

    // From OpenDSA Data Structures and Algorithms, Chapter 13,
    // https://opendsa-server.cs.vt.edu/OpenDSA/Books/Everything/html/InsertionSort.html
    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) {
                swap(A, j, j - 1);
            }
        }
    }

• At the beginning of each outer iteration, items 0 … i-1 are properly ordered.

• Each outer iteration seeks to insert item A[i] into the appropriate position within 0 … i.

2 Insertion Sort: Worst Case Analysis

• The inner loop body is O(1).

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) {
                swap(A, j, j - 1);                              // O(1)
            }
        }
    }

Looking at the inner loop,

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) {
                swap(A, j, j - 1);                              // O(1)
            }
        }
    }

Question: In the worst case, how many times do we go around the inner loop (to within plus or minus 1)?

• 0 times

• 1 time

• i times

• j times

• n times

• A.length times

With that determined,

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) { // cond: O(1) #: i
                swap(A, j, j - 1);                              // O(1)
            }
        }
    }

Moving on…

Question: So what is the complexity of the inner loop?

• O(1)

• O(i)

• O(j)

• O(A.length)

• None of the above

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) { // cond: O(1) #: i total: O(i)
                swap(A, j, j - 1);                              // O(1)
            }
        }
    }

Now, looking at the outer loop body, the entire outer loop body is $O(i)$.

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // cond: O(1) # A.length
            O(i)            }
    }

Let $n$ denote A.length. The outer loop executes $n-1$ times.

Question: What, then is the complexity of the entire outer loop?

• $O(i)$

• $O(n)$

• $O(i*(n-1))$

• $O(i*n)$

• $O(i^2)$

• $O(n^2)$

• $O((n*(n-1))/2)$

If you gave any answer involving i, you should have known better from the start. Complexity of a block of code must always be described in therms of the inputs to that code. i is not an input to the loop - any value it might have held prior to the start of the loop is ignored and overwritten.

Question: What, then is the complexity of the entire function?

• $O(n)$

• $O(n^2)$

• $O((n*(n-1))/2)$

• None of the above

Insertion sort has a worst case of $O(N^2)$ where $N$ is the size of the input list.

3 Insertion Sort: special cases

As a special case, consider the behavior of this algorithm when applied to an array that is already sorted.

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) {
                swap(A, j, j - 1);
            }
        }
    }

• Note that if the array is already sorted, then we never enter the body of the inner loop. The inner loop is then $O(1)$ and inssort is $O(\mbox{A.length})$.

This makes insertion sort a reasonable choice when adding a few items to a large, already sorted array.

Suppose that an array is is -almost_ sorted – i.e., no element is more than $k$ positions out of place, where $k$ is some (small) constant.

We proceed mch as in the earlier analysis, but if not element needs to be moved more the $k$ positions, then the inner loop will never repeat more than $k$ times.

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) { // cond: O(1) #: k total: O(k)
                swap(A, j, j - 1);         // O(1)
            }
        }
    }

The inner loop is therefore $O(k)$. Moving to the outer loop,

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // cond: O(1)  #: A.length  total: O(k*A.length)
            // O(k)
        }
    }

The total complexity of the loop is therefore $O(k * \mbox{A.length})$. But we defined $k$ as a constant, so this actually simplifies to $O(k\mbox{A.length})$!

This makes insertion sort a reasonable choice when an arrays is “almost” sorted.

We’ll use this special case in other lesson, quite soon, as a way to speed up a more elaborate sorting algorithm.

4 Average-Case Analysis for Insertion Sort

Instead of doing the average case analysis by the copy-and-paste technique, we’ll produce a result that works for all algorithms that behave like it.

Define an inversion of an array a as any pair (i,j) such that i<j but a[i]>a[j].

Question: How many inversions in this array?

$[29 \; 10 \; 14 \; 37 \; 13]$

4.1 Inversions

In an array of n elements, the most inversions occur when the array is in exactly reversed order. Inversions then are

Counting these we have (starting from the bottom): $\sum_{i=1}^{n-1} i$ inversions. So the total # of inversions is $\frac{n*(n-1)}{2}$.

We’ll state this formally:

Theorem: The maximum number of inversions in an array of $n$ elements is $(n*(n-1))/2$.

We have just proven that theorem. Now, another one, describing the average:

Theorem: The average number of inversions in an array of $n$ randomly selected elements is $(n*(n-1))/4$.

We won’t prove this, but note that it makes sense, since the minimum number of inversions is 0, and the maximum is $(n*(n-1))/2$, so it makes intuitive sense that the average would be the midpoint of these two values.

4.2 A Speed Limit on Adjacent-Swap Sorting

Now, the result we have been working toward:

Theorem: Any sorting algorithm that only swaps adjacent elements has average time no faster than $O(n^2)$.

Proof

Swapping 2 adjacent elements in an array removes at most 1 inversion.

But on average, there are $O(n^2)$ inversions, so the total number of swaps required is at least $O(n^2)$.

Hence the algorithm as a whole can be no faster than $O(n^2)$.

QED

And,

Corollary: Insertion sort has average case complexity $O(n^2)$.

Proof

Insertion sort is written like this:

    public static <T extends Comparable<T>> void inssort(T[] A) {
        for (int i = 1; i < A.length; i++) { // Insert i'th record
            for (int j = i; (j > 0) && (A[j].compareTo(A[j - 1]) < 0); j--) {
                swap(A, j, j - 1);
            }
        }
    }

and it is clear that this version only exchanges adjacent elements.

By the theorem just given, the best average case complexity we could therefore get is $O(n^2)$.

The theorem does not preclude an average case complexity even slower than that, but we know that the worst case complexity is also $O(n^2)$, and the average case can’t be any slower than the worst case.

So we conclude that the average case complexity is, indeed, $O(n^2)$.