Sorting --- Quick Sort
Steven J. Zeil
Quick sort is a sorting algorithm with the optimal O(nlogn) average case, but a O(n2) worst case.
Despite its slower worst case behavior, the Quick sort is often the preferred approach when sorting array-like structures. This is because the constant multiplier on quick sort’s average case is much smaller, and, if we are careful, we can make the worst case input of quick sort a relatively obscure case that would seldom be seen in practice.
Like Merge sort, Quick sort also works by dividing the array into progressively smaller parts, but instead of chopping at an arbitrary point (e.g., halfway), it divides into pieces in which
-
all items in the left section are less than or equal to a pivot value,
-
all items on the right are greater than the pivot value.
This is often called partitioning the array.
For example, given the array
1 | 4 | 3 | 9 | 5 | 8 | 7 | 6 |
it’s not hard to see that we can split the array this way
1 | 4 | 3 |
9 | 5 | 8 | 7 | 6 |
so that all items to the left are less than or equal to a pivot value 4, and all items on the right are greater than 4.
On the other hand, for some arrays:
9 | 1 | 8 | 7 | 2 | 6 | 4 | 0 |
there’s really no place were we can divide the array “as is” to meet these requirements.
So an algorithm to partition the array must, in some cases, move elements around to make the partition possible. The partition algorithm from your text does just this.
1 Partitioning the Array
static <T extends Comparable<T>> int partition(T[] A, int left, int right, T pivot) {
while (left <= right) { // Move bounds inward until they meet
while (A[left].compareTo(pivot) < 0) {
left++;
}
while ((right >= left) && (A[right].compareTo(pivot) >= 0)) {
right--;
}
if (right > left) {
swap(A, left, right);
} // Swap out-of-place values
}
return left; // Return first position in right partition
}
This assumes that we have already chosen a value to serve as the pivot
element.
Typical choices are to use the first element in the portion of the array being partitioned, or the last element in that portion of the array, or to use the middle value from that portion of the array. We’ll discuss these options later.
The partition algorithm basically starts with the left and right indices at opposite ends of the portion of the array being operated on. These are moved towards each other until they meet.
left is moved up first, until it either meets right or hits an element greater than or equal to the pivot
.
Then right is moved down until it hits an element less than or equal to the pivot.
If left is pointing to an element greater than the pivot, and right is pointing to an element less than the pivot, those two elements are clearly out of order with respect to each other. Clearly, we will never find a place to partition the array as long as we have these two elements, the smaller one coming after the larger one. So, we swap the two, and then resume moving left and right towards each other again.
Run this algorithm as an animation until you are comfortable with your understanding of how it works.
1.1 Analysis of partition
Let’s do the analysis of this routine.
The loop bodies of the two while loops are clearly O(1).
static <T extends Comparable<T>> int partition(T[] A, int left, int right, T pivot) {
while (left <= right) { // Move bounds inward until they meet
while (A[left].compareTo(pivot) < 0) {
left++; // O(1)
}
while ((right >= left) && (A[right].compareTo(pivot) >= 0)) {
right--; // O(1)
}
if (right > left) {
swap(A, left, right);
} // Swap out-of-place values
}
return left; // Return first position in right partition
}
So are the while loop conditions.
static <T extends Comparable<T>> int partition(T[] A, int left, int right, T pivot) {
while (left <= right) { // Move bounds inward until they meet
while (A[left].compareTo(pivot) < 0) { // cond: O(1)
left++; // O(1)
}
while ((right >= left) && (A[right].compareTo(pivot) >= 0)) { // cond: O(1)
right--; // O(1)
}
if (right > left) {
swap(A, left, right);
} // Swap out-of-place values
}
return left; // Return first position in right partition
}
Question: How many times does each while loop execute (in the worst case)?
-
1
-
n
-
right
-
left
-
right-left
-
right+left
static <T extends Comparable<T>> int partition(T[] A, int left, int right, T pivot) {
while (left <= right) { // Move bounds inward until they meet
while (A[left].compareTo(pivot) < 0) { // cond: O(1) # (right-left)
left++; // O(1)
}
while ((right >= left) && (A[right].compareTo(pivot) >= 0)) { // cond: O(1) # (right-left)
right--; // O(1)
}
if (right > left) {
swap(A, left, right);
} // Swap out-of-place values
}
return left; // Return first position in right partition
}
So the while loops are each O(right-left).
static <T extends Comparable<T>> int partition(T[] A, int left, int right, T pivot) {
while (left <= right) { // Move bounds inward until they meet
// # O(right-left)
// # O(right-left)
if (right > left) {
swap(A, left, right);
} // Swap out-of-place values
}
return left; // Return first position in right partition
}
So the body of the for loop is clearly O(right−left). How many times does this loop repeat? It’s hard to say, but we don’t actually need an exact count. Each time the two inner loops were executed, they may move right and left closer to one another. In addition, the final increments in the for loop move those variables still closer to one another. The total number of times the for loop body gets executed can be no more than right-left
times, and the total number of times both while loop bodies can be executed is also right-left
.
Of course, those loops also change the values of right and left, so it is more accurate to say that the total number of executions of those loops is the initial value of right-left
.
So the loop (and the partition algorithm) are O(right−left). In other words, this algorithm is linear in the number of elements to be partitioned.
2 Quick Sort
To actually sort data, we use the partition function within the quicksort routine shown here.
public static <T extends Comparable<T>> void quicksort(T[] A) {
quicksort(A, 0, A.length - 1); // Kick off the recursion
}
// based on OpenDSA Data Structures and Algorithms, Chapter 13,
// https://opendsa-server.cs.vt.edu/OpenDSA/Books/Everything/html/Quicksort.html
private static <T extends Comparable<T>> void quicksort(T[] A, int i, int j) { // Quicksort
if (i < j + 1) {
int pivotindex = findpivot(A, i, j); // Pick a pivot
swap(A, pivotindex, j); // Stick pivot at end
// k will be the first position in the right subarray
int k = partition(A, i, j - 1, A[j]);
swap(A, k, j); // Put pivot in place
quicksort(A, i, k - 1); // Sort left partition
quicksort(A, k + 1, j); // Sort right partition
}
}
The recursive routine is designed to sort all elements of an array from position i to j, inclusively.
-
First, we check to be sure this is not a “degenerate” case of 0 or 1 elements. An array of 0 elements or of 1 element is already sorted, and we can return immediately.
- The textbook omits this step, but it significantly improves the performance of quicksort.
-
Second, we use the partition function to partition the array.
-
At the end of the previous step, we know that all of the elements on the left are less than any element on the right. Consequently, if we could sort the left part and the right part separately, the array as a whole would be sorted.
And so, we accomplish that separate sorting of the left and right parts by recursively calling quicksort on each of the two parts.
Run this algorithm as an animation until you are comfortable with your understanding of how it works.
3 Quick Sort Analysis
3.1 Best Case
private static <T extends Comparable<T>> void quicksort(T[] A, int i, int j) { // Quicksort
if (i < j + 1) {
int pivotindex = findpivot(A, i, j); // Pick a pivot
swap(A, pivotindex, j); // Stick pivot at end
// k will be the first position in the right subarray
int k = partition(A, i, j - 1, A[j]);
swap(A, k, j); // Put pivot in place
quicksort(A, i, k - 1); // Sort left partition
quicksort(A, k + 1, j); // Sort right partition
}
}
We don’t often do a “best case” analysis, but it might help to understand the behavior of this particular algorithm.
Ideally, Each call to partition will divide the array exactly in half. Thus each recursive call will work on exactly half of the array.
partition, we have determined, is O(last−first). Suppose we start with N items. Partitioning this via partition takes O(N) time.
Then, in this best case, we would have two sub-arrays, each with N/2 elements. Eventually, each of these will be partitioned at a cost of O(N/2)+O(N/2)=O(N) effort.
Each of those sub-arrays will, upon partitioning, yield a pair of sub-arrays of size N/4. So, we will, over the course of the entire algorithm, be asked 4 times to partition sub-arrays of size N/4, for a total cost of 4∗O(N/4)=O(N).
You can see what’s happening. In general, at the kth level of recursion, there will be 2k sub-arrays of size N/(2k) to be pivoted, at a total cost of ∑2ki=1O(N/(2k))=O(N). This continues until N/(2k) has been reduced to 1.
So all calls at on sub-arrays of the same size add up to O(N). There are log(N) levels, so the best case is O(Nlog(N)).
3.2 Average Case
The average case for quick sort is O(Nlog(N)). The proof of is rather heavy going, but is presented in the final section for those who are interested.
3.3 Worst Case
In the worst case, each call to partition
would divide the array with one element on one side and all of the other elements on the other side. Then, when we do the recursive calls, one call returns immediately but the other has almost as much work left to do as before we did the pivot. The recursive calls go N−1 levels deep, each applying a linear time pivot to its part of the array. Hence we get a total effort from these calls of
O(N−1)+O(N−2)+…+O(1)=O(n−1∑i=1i)=O(N2)
3.4 Great Average, Dreadful Worst
Quick sort therefore poses an interesting dilemma. Its average case is very fast. In addition, its multiplicative constant is also fairly low, so it tends to be the fastest, on average, of the known O(NlogN) average-case sorting algorithms in actual clock-time. But its worst case is just dreadful.
So the suitability of quick sort winds up coming down to a question of how often we would actually expect to encounter the worst-case or near-worst-case behavior. That, in turn, depends upon the choice of pivot.
3.4.1 Choosing the Pivot
The code shown here selects the midpoint of the range as the pivot location.
The choice of pivot value is made by the findPivot
function. Some options for this would be:
static <T extends Comparable<T>> int findpivot(T[] A, int i, int j) {
return i; // choose the first element
}
static <T extends Comparable<T>> int findpivot(T[] A, int i, int j) {
return j; // choose the last element
}
static <T extends Comparable<T>> int findpivot(T[] A, int i, int j) {
return (i + j) / 2; // choose the middle element
}
The first two turn out to be particularly bad choices, as the worst case would then occur whenever the array was already sorted or was in exactly reverse order. Those are, in practice, very common situations.
Choosing the midpoint is somewhat better. The worst case is somewhat more obscure and far less likely in practice. It’s not entirely unlikely however – some tree traversal algorithms that we will covered in later lessons could give rise to listings of elements that would have the largest/smallest elements in each range at the midpoint, causing the O(N2) behavior.
In practice then, the most commonly recommended choices seem to be:
-
Select a position in the range to be sorted at random.
This doesn’t change the fact that the worst case exists and is O(N2), but it reduces the probability of actually seeing the worst case and, because of the randomness, dramatically reduces the chances that any given application would generate patterns of data that would trigger the worst-case many times.
-
Examine the first, last, and midpoint values in the range to be sorted. Choose for the pivot whichever of these three values lies between the other two (in value, not position). This is called the median-of-three rule.
The idea here is that we know that the pivot is neither the smallest nor largest value (though it could be the next-to-largest or next-to-smallest). Sorted arrays and reverse-sorted arrays will actually yield best-case behavior (because the mid-point value will be used in these cases). The O(N2) worst case still exists, but it’s not a common input pattern.
4 Optimizing the Algorithm
If you have run the quicksort as an animation, you may have noticed that a lot of its work is spent on very small sub-arrays. These small arrays are also responsible for a good portion of the storage overhead of quicksort — space on the run-time stack.
A significant speedup comes from only applying quick sort to fairly large arrays:
public static <T extends Comparable<T>> void quicksort(T[] A) {
quicksort(A, 0, A.length - 1); // Sort everything into almost correct position
inssort(A); // "clean up" with an insertion sort
}
// based on OpenDSA Data Structures and Algorithms, Chapter 13,
// https://opendsa-server.cs.vt.edu/OpenDSA/Books/Everything/html/Quicksort.html
private static <T extends Comparable<T>> void quicksort(T[] A, int i, int j) { // Quicksort
if (i < j + 10) {
int pivotindex = findpivot(A, i, j); // Pick a pivot
swap(A, pivotindex, j); // Stick pivot at end
// k will be the first position in the right subarray
int k = partition(A, i, j - 1, A[j]);
swap(A, k, j); // Put pivot in place
quicksort(A, i, k - 1); // Sort left partition
quicksort(A, k + 1, j); // Sort right partition
}
}
-
This test used to be
if (i < j + 1)
, so that we would quicksort any array of 2 or more elements. Now we change it to only quick sort arrays of more than 10 elements.With this change, our recursive quick sort only guarantees that each element will be within 10 places of where it should be.
-
Then we do a final “cleanup” by applying inssort (insertion sort) to the array after all the recursion is done.
Although inssort is O(N2) in its worst case, in the special case where all elements are no more than k places away from their final position, insertionSort is O(k∗N).
In this situation, we know that the recursive quick sort has brought each element to within 10 positions of its sorted position, so this insertionSort pass is O(10∗N)=O(N). Since we’ve already expended an average of O(Nlog(N)) work before that pass, this cleanup phase is still efficient by comparison.
5 Iterative Quicksort
We have noted previously that many recursive algorithms can be converted to iterative forms by the use of a stack. quicksort
is actually a good example of this.
Look at the actually recursive calls in quicksort
:
private static <T extends Comparable<T>> void quicksort(T[] A, int i, int j) { // Quicksort
⋮
quicksort(A, i, k - 1); // Sort left partition
quicksort(A, k + 1, j); // Sort right partition
}
}
We can see that all that changes, from one call to another, are the two integer values denoting the starting and ending range of the portion of the array that we want to sort.
So, if we introduce a data type to hold such a pair of numbers:
private static class Range {
public int left;
public int right;
public Range(int left0, int right0) {
left = left0;
right = right0;
}
}
then we can simulate recursion by using a stack of ranges:
public static <T extends Comparable<T>> void quicksort2(T[] A) {
Stack<Range> stack = new Stack<>();
stack.push(new Range(0, A.length - 1));
while (!stack.isEmpty()) {
Range range = stack.pop();
int i = range.left;
int j = range.right;
⋮
// Sort everything in the range [i..j]
⋮
}
}
To fill in the missing portion in the middle, we simply copy-and-paste the body of our old recursive function:
public static <T extends Comparable<T>> void quicksort2(T[] A) {
Stack<Range> stack = new Stack<>();
stack.push(new Range(0, A.length - 1));
while (!stack.isEmpty()) {
Range range = stack.pop();
int i = range.left;
int j = range.right;
if (i < j + 10) {
int pivotindex = findpivot(A, i, j); // Pick a pivot
swap(A, pivotindex, j); // Stick pivot at end
// k will be the first position in the right subarray
int k = partition(A, i, j - 1, A[j]);
swap(A, k, j); // Put pivot in place
quicksort(A, i, k - 1); // Sort left partition
quicksort(A, k + 1, j); // Sort right partition
}
}
}
and then replace the recursive calls in that block of code by pushes onto the stack:
public static <T extends Comparable<T>> void quicksort2(T[] A) {
Stack<Range> stack = new Stack<>();
stack.push(new Range(0, A.length - 1));
while (!stack.isEmpty()) {
Range range = stack.pop();
int i = range.left;
int j = range.right;
if (i < j + 10) {
int pivotindex = findpivot(A, i, j); // Pick a pivot
swap(A, pivotindex, j); // Stick pivot at end
// k will be the first position in the right subarray
int k = partition(A, i, j - 1, A[j]);
swap(A, k, j); // Put pivot in place
stack.push(new Range(i, k-1)); // Sort left partition
stack.push(new Range(k+1, j)); // Sort right partition
}
}
}
So, each time around the while1
loop, we simply pick up the starting and ending positions of a sub-array that needs to be partitioned and sorted.
Run this algorithm as an animation until you are comfortable with your understanding of how it works.
This iterative form, when used with a reasonable pivot rule (such as median-of-3), is not only O(N log N) on average, but has an extremely low multiplicative constant, making it one of the best choices as a practical sorting algorithm.
6 Average Case Analysis of Quick Sort
The average case run time of Quick sort is O(nlogn). Proving this is really a bit much for this course, but we present the proof here for the curious:
-
Assume a strategy for choosing pivots such that, after partitioning A into A1 and A2, all lengths of A1 from 0 to |A|−1 are equally likely.
-
The running time of quickSort equals the time of its two recursive calls plus time to do the partition.
-
partition is O(last−first).
Suppose that we partition n elements into sub-arrays of length i and (n−i).
-
Time T to sort the n elements is then:
T(n)=T(i)+T(n−i)+c∗n
- This kind of formula is called a recurrence relation. They are very common in describing the performance of recursive routines.
Because i is equally likely to be any value from 0 to n−1, the average (expected) value of T(i) is
E(T(i))=1/nn−1∑j=0T(j)
Since n−i can take on the same values as i, and all such values are equally likely,
E(T(n−i))=E(T(i))
On average, then
T(n)=2/n(n−1∑j=0T(j))+c∗n
Multiply through by n:
n∗T(n)=2∗(n−1∑j=0T(j))+c∗n2
n is just a variable, so this must be true no matter what value we plug in for it. Try replacing n by n−1.
(n−1)∗T(n−1)=2∗(n−2∑j=0T(j))+c∗(n−1)2
So now both of these are true:
n∗T(n)=2∗(n−1∑j=0T(j))+c∗n2(n−1)∗T(n−1)=2∗(n−2∑j=0T(j))+c∗(n−1)2
Subtract the 2nd equation from the first, obtaining: nT(n)−(n−1)T(n−1)=2T(n−1)+2cn−c
Collect the T(n−1) terms together and drop the −c term: nT(n)=(n+1)T(n−1)+2cn
Next we apply a standard technique for solving recurrence relations. Divide by n(n+1) and “telescope”: T(n)n+1=T(n−1)n+2cn+1T(n−1)n=T(n−2)n−1+2cnT(n−2)n−1=T(n−3)n−2+2cn−1⋮
Note that most of the terms on the left will have appeared on the right in the previous equation, so if we were to add up all these equations, these terms would appear on both sides and could be dropped: T(n)n+1=T(1)2+2cn+1∑j=31/j
As n gets very large, ∑n+1j=31/j approaches ln(n)+γ, where γ is Euler’s constant, 0.577….
Hence T(n)n+1=T(1)2+2c∗ln(n)+2cγ=ln(n)+c2=O(ln(n))
and so T(n) is O(nlog(n)).