Sorting --- Quick Sort
Steven J. Zeil
Quick sort is a sorting algorithm with the optimal $O(n \log n)$ average case, but a $O(n^2)$ worst case.
Despite its slower worst case behavior, the Quick sort is generally preferred over Merge sort when sorting array-like structures (including vectors and deques). This is because the constant multiplier on quick sort’s average case is much smaller, and, if we are careful, we can make the worst case input of quick sort a relatively obscure case that would seldom be seen in practice.
Like Merge sort, Quick sort also works by dividing the array into progressively smaller parts, but instead of chopping at an arbitrary point (e.g., halfway), it divides into pieces in which
-
all items in the left section are less than or equal to a pivot value,
-
all items on the right are greater than the pivot value.
This is often called partitioning the array.
For example, given the array
| 1 | 4 | 3 | 9 | 5 | 8 | 7 | 6 |
it’s not hard to see that we can split the array this way
| 1 | 4 | 3 |
| 9 | 5 | 8 | 7 | 6 |
so that all items to the left are less than or equal to a pivot value 4, and all items on the right are greater than 4.
On the other hand, for some arrays:
| 9 | 1 | 8 | 7 | 2 | 6 | 4 | 0 |
there’s really no place were we can divide the array “as is” to meet these requirements.
So an algorithm to partition the array must, in some cases, move elements around to make the partition possible. The pivotIndex algorithm from your text does just this.
1 Partitioning the Array
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
// move up lower sublist; stop when scanUp enters
// upper sublist or identifies an element >= pivot
while (scanUp <= scanDown && v[scanUp] < pivot)
scanUp++;
// scan down upper sublist; stop when scanDown locates
// an element <= pivot; we guarantee we stop at arr[first]
while (pivot < v[scanDown])
scanDown--;
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
First, we choose a value to serve as the pivot element. In this version, we are using the midpoint of the data sequence. There are other possibilities, some of which will be discussed later.
The pivotIndex algorithm basically starts with the scanUp and scanDown indices at opposite ends of the array. These are moved towards each other until they meet.
scanUp is moved up first, until it either meets scanDown or hits an element greater than or equal to the pivot.
Then scanDown is moved down until it hits an element less than or equal to the pivot.
If scanUp is pointing to an element greater than the pivot, and scanDown is pointing to an element less than the pivot, those two elements are clearly out of order with respect to each other. Clearly, we will never find a place to partition the array as long as we have these two elements, the smaller one coming after the larger one. So, we swap the two, and then resume moving scanUp and scanDown towards each other again.
Try out the quicksort in an animation. Run it until the end ofthe first partitioning, watching how elements are swapped between the left nad right sections.
1.1 Analysis of pivotIndex
Let’s do the analysis of this routine.
The loop bodies of the two while loops are clearly O(1).
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
// move up lower sublist; stop when scanUp enters
// upper sublist or identifies an element >= pivot
while (scanUp <= scanDown && v[scanUp] < pivot)
scanUp++; // O(1)
// scan down upper sublist; stop when scanDown locates
// an element <= pivot; we guarantee we stop at arr[first]
while (pivot < v[scanDown])
scanDown--; // O(1)
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
So are the while loop conditions.
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
// move up lower sublist; stop when scanUp enters
// upper sublist or identifies an element >= pivot
while (scanUp <= scanDown && v[scanUp] < pivot) // O(1)
scanUp++; // O(1)
// scan down upper sublist; stop when scanDown locates
// an element <= pivot; we guarantee we stop at arr[first]
while (pivot < v[scanDown]) // O(1)
scanDown--; // O(1)
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
Question: How many times does each while loop execute (in the worst case)?
-
1
-
n
-
scanDown
-
scanUp
-
scanDown-scanUp
-
scanDown+scanUp
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
// move up lower sublist; stop when scanUp enters
// upper sublist or identifies an element >= pivot
while (scanUp <= scanDown && v[scanUp] < pivot) // cond: O(1) #: (scanDown-scanUp)
scanUp++; // O(1)
// scan down upper sublist; stop when scanDown locates
// an element <= pivot; we guarantee we stop at arr[first]
while (pivot < v[scanDown]) // cond: O(1) #: (scanDown-scanUp)
scanDown--; // O(1)
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
So the while loops are each O(scanDown-scanUp).
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
O(scanDown-scanUp)
O(scanDown-scanUp)
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
So the body of the for loop is clearly $O(\mbox{scanDown}-\mbox{scanUp})$. How many times does this loop repeat? It’s hard to say, but we don’t actually need an exact count. Each time the two inner loops were executed, they may move scanDown and scanUp closer to one another. In addition, the final increments in the for loop move those variables still closer to one another. The total number of times the for loop body gets executed can be no more than scanDown-scanUp times, and the total number of times both while loop bodies can be executed is also scanDown-scanUp.
Of course, those loops also change the values of scanDown and scanUp, so it is more accurate to say that the total number of executions of those loops is the initial value of scanDown-scanUp.
Looking at the initialization before the loop, we see that these initial values are last-1 and first+1, respectively. So the loop (and the pivotIndex algorithm) are $O(\mbox{last}-\mbox{first})$. In other words, this algorithm is linear in the number of elements to be partitioned.
2 Quick Sort
To actually sort data, we use the pivotIndex function within the quicksort routine shown here.
template <typename T>
void quicksort(vector<T>& v, int first, int last)
{
// index of the pivot
int pivotLoc;
// temp used for an exchange when [first,last) has
// two elements
T temp;
// if the range is not at least two elements, return
if (last - first <= 1)
return;
// if sublist has two elements, compare v[first] and
// v[last-1] and exchange if necessary
else if (last - first == 2)
{
if (v[last-1] < v[first])
{
temp = v[last-1];
v[last-1] = v[first];
v[first] = temp;
}
return;
}
else
{
pivotLoc = pivotIndex(v, first, last);
// make the recursive call
quicksort(v, first, pivotLoc);
// make the recursive call
quicksort(v, pivotLoc +1, last);
}
}
The recursive routine is designed to sort all elements of an array from position low to high, inclusively.
-
First, we check to be sure this is not a “degenerate” case of 0, 1, or 2 elements. We can resolve those cases immediately.
-
Second, we use the pivotIndex function to partition the array.
-
At the end of the previous step, we know that all of the elements on the left are less than any element on the right. Consequently, if we could sort the left part and the right part separately, the array as a whole would be sorted.
And so, we accomplish that separate sorting of the left and right parts by recursively calling quicksort on each of the two parts.
Try out the full quicksort algorithm in an animation.
3 Quick Sort Analysis
3.1 Best Case
template <typename T>
void quicksort(vector<T>& v, int first, int last)
{
// index of the pivot
int pivotLoc;
// temp used for an exchange when [first,last) has
// two elements
T temp;
// if the range is not at least two elements, return
if (last - first <= 1)
return;
// if sublist has two elements, compare v[first] and
// v[last-1] and exchange if necessary
else if (last - first == 2)
{
if (v[last-1] < v[first])
{
temp = v[last-1];
v[last-1] = v[first];
v[first] = temp;
}
return;
}
else
{
pivotLoc = pivotIndex(v, first, last);
// make the recursive call
quicksort(v, first, pivotLoc);
// make the recursive call
quicksort(v, pivotLoc +1, last);
}
}
We don’t often do a “best case” analysis, but it might help to understand the behavior of this particular algorithm.
Ideally, Each call to pivotIndex will divide the array exactly in half. Thus each recursive call will work on exactly half of the array.
pivotIndex, we have determined, is $O(\mbox{last}-\mbox{first})$. Suppose we start with $N$ items. Partitioning this via pivotIndex takes $O(N)$ time.
Then, in this best case, we would have two sub-arrays, each with $N/2$ elements. Eventually, each of these will be partitioned at a cost of $O(N/2) + O(N/2) = O(N)$ effort.
Each of those sub-arrays will, upon partitioning, yield a pair of sub-arrays of size $N/4$. So, we will, over the course of the entire algorithm, be asked 4 times to partition sub-arrays of size $N/4$, for a total cost of $4*O(N/4) = O(N)$.
You can see what’s happening. In general, at the $k^{\mbox{th}}$ level of recursion, there will be $2^k$ sub-arrays of size $N/(2^k)$ to be pivoted, at a total cost of $\sum_{i=1}^{2^k} O(N/(2^k)) = O(N)$. This continues until $N/(2^k)$ has been reduced to 1.
So all calls at on sub-arrays of the same size add up to $O(N)$. There are $\log(N)$ levels, so the best case is $O(N \log(N))$.
3.2 Average Case
The average case for quick sort is $O(N \, log(N))$. The proof of is rather heavy going, but is presented in the final section for those who are interested.
3.3 Worst Case
In the worst case, each call to pivotIndex would divide the array with one element on one side and all of the other elements on the other side. Then, when we do the recursive calls, one call returns immediately but the other has almost as much work left to do as before we did the pivot. The recursive calls go $N-1$ levels deep, each applying a linear time pivot to its part of the array. Hence we get a total effort from these calls of
\[ O(N-1) + O(N-2) + … + O(1) = O\left(\sum_{i=1}^{n-1} i \right) = O(N^2) \]
3.4 Great Average, Dreadful Worst
Quick sort therefore poses an interesting dilemma. Its average case is very fast. In addition, its multiplicative constant is also fairly low, so it tends to be the fastest, on average, of the known $O(N \, \log N)$ average-case sorting algorithms in actual clock-time. But its worst case is just dreadful.
So the suitability of quick sort winds up coming down to a question of how often we would actually expect to encounter the worst-case or near-worst-case behavior. That, in turn, depends upon the choice of pivot.
3.4.1 Choosing the Pivot
The code shown here selects the midpoint of the range as the pivot location.
template <typename T>
int pivotIndex(vector<T>& v, int first, int last)
{
// index for the midpoint of [first,last) and the
// indices that scan the index range in tandem
int mid, scanUp, scanDown;
// pivot value and object used for exchanges
T pivot, temp;
if (first == last)
return last;
else if (first == last-1)
return first;
else
{
mid = (last + first)/2;
pivot = v[mid];
// exchange the pivot and the low end of the range
// and initialize the indices scanUp and scanDown.
v[mid] = v[first];
v[first] = pivot;
scanUp = first + 1;
scanDown = last - 1;
// manage the indices to locate elements that are in
// the wrong sublist; stop when scanDown <= scanUp
for(;;)
{
// move up lower sublist; stop when scanUp enters
// upper sublist or identifies an element >= pivot
while (scanUp <= scanDown && v[scanUp] < pivot)
scanUp++;
// scan down upper sublist; stop when scanDown locates
// an element <= pivot; we guarantee we stop at arr[first]
while (pivot < v[scanDown])
scanDown--;
// if indices are not in their sublists, partition complete
if (scanUp >= scanDown)
break;
// indices are still in their sublists and identify
// two elements in wrong sublists. exchange
swap(v[scanUp], v[scanDown]);
scanUp++;
scanDown--;
}
// copy pivot to index (scanDown) that partitions sublists
// and return scanDown
v[first] = v[scanDown];
v[scanDown] = pivot;
return scanDown;
}
}
Other choices that were once popular included choosing the first element or the last element of the range to serve as the pivot. You can see that, once we have chosen a pivot, the first thing we do is to move it “out of the way” by swapping it with the first element. So, if we simply chose the first [or last] element as the pivot, it would already be out of the way, and we could save that step.
These turned out to be poor choices, however, as the worst case would then occur whenever the array was already sorted or was in exactly reverse order. Those are, in practice, very common situations.
Choosing the midpoint is somewhat better. The worst case is somewhat more obscure and far less likely in practice. It’s not entirely unlikely however – some tree traversal algorithms that we will covered in later lessons could give rise to listings of elements that would have the largest/smallest elements in each range at the midpoint, causing the $O(N^2)$ behavior.
In practice then, the most commonly recommended choices seem to be:
-
Select a position in the range to be sorted at random.
This doesn’t change the fact that the worst case exists and is $O(N^2)$, but it reduces the probability of actually seeing the worst case and, because of the randomness, dramatically reduces the chances that any given application would generate patterns of data that would trigger the worst-case many times.
-
Examine the first, last, and midpoint values in the range to be sorted. Choose for the pivot whichever of these three values lies between the other two (in value, not position). This is called the median-of-three rule.
The idea here is that we know that the pivot is neither the smallest nor largest value (though it could be the next-to-largest or next-to-smallest). Sorted arrays and reverse-sorted arrays will actually yield best-case behavior (because the mid-point value will be used in these cases). The $O(N^2)$ worst case still exists, but it’s not a common input pattern.
4 Optimizing the Algorithm
If you have run the quick sort, you may have noticed that a lot of its work is spent on very small sub-arrays. These small arrays are also responsible for a good portion of the storage overhead of Quick sort — space on the run-time stack.
A significant speedup comes from only applying quick sort to fairly large arrays:
template <typename T>
void quicksort(vector<T>& v, int first, int last)
{
// index of the pivot
int pivotLoc;
// temp used for an exchange when [first,last) has
// two elements
T temp;
// if the range is not at least ten elements, return
if (last - first <= 10)
return;
// if sublist has two elements, compare v[first] and
// v[last-1] and exchange if necessary
else if (last - first == 2)
{
if (v[last-1] < v[first])
{
temp = v[last-1];
v[last-1] = v[first];
v[first] = temp;
}
return;
}
else
{
pivotLoc = pivotIndex(v, first, last);
// make the recursive call
quicksort(v, first, pivotLoc);
// make the recursive call
quicksort(v, pivotLoc +1, last);
}
}
template <typename T>
void quicksortR(vector<T>& v, int first, int last)
{
// index of the pivot
int pivotLoc;
// temp used for an exchange when [first,last) has
// two elements
T temp;
// if the range is not at least two elements, return
if (last - first <= 10)
return;
// if sublist has two elements, compare v[first] and
// v[last-1] and exchange if necessary
else if (last - first == 2)
{
if (v[last-1] < v[first])
{
temp = v[last-1];
v[last-1] = v[first];
v[first] = temp;
}
return;
}
else
{
pivotLoc = pivotIndex(v, first, last);
// make the recursive call
quicksort(v, first, pivotLoc);
// make the recursive call
quicksort(v, pivotLoc +1, last);
}
}
void quicksort(vector<T>& v, int first, int last)
{
quicksortR (v, first, last);
insertionSort (v, first, last);
}
-
This test used to be
if (last - first >= 1), so that we would quick sort any array of 2 or more elements. Now we change it to only quick sort arrays of more than 10 elements.With this change, our recursive quick sort only guarantees that each element will be within 10 places of where it should be.
-
Then we do a final “cleanup” by applying insertionSort to the array after all the recursion is done.
Although insertionSort is $O(N^2)$ in its worst case, in the special case where all elements are no more than $k$ places away from their final position, insertionSort is $O(k*N)$.
In this situation, we know that the recursive quick sort has brought each element to within $10$ positions of its sorted position, so this insertionSort pass is $O(10*N) = O(N)$. Since we’ve already expended an average of $O(N \, \log(N))$ work before that pass, this cleanup phase is still efficient by comparison.
5 Average Case Analysis of Quick Sort
The average case run time of Quick sort is $O(n \, \log n)$. Proving this is really a bit much for this course, but we present the proof here for the curious:
-
Assume a strategy for choosing pivots such that, after partitioning A into A1 and A2, all lengths of A1 from $0$ to $|A|-1$ are equally likely.
-
The running time of quickSort equals the time of its two recursive calls plus time to do the partition.
-
pivotIndex is $O(\mbox{last}-\mbox{first})$.
Suppose that we partition $n$ elements into sub-arrays of length $i$ and $(n-i)$.
-
Time $T$ to sort the $n$ elements is then:
\[ T(n) = T(i)+T(n-i)+c*n \]
- This kind of formula is called a recurrence relation. They are very common in describing the performance of recursive routines.
Because $i$ is equally likely to be any value from $0$ to $n-1$, the average (expected) value of $T(i)$ is
\[E(T(i)) = 1/n \sum_{j=0}^{n-1} T(j) \]
Since $n-i$ can take on the same values as $i$, and all such values are equally likely,
\[ E(T(n-i))=E(T(i)) \]
On average, then
\[ T(n) = 2/n \left(\sum_{j=0}^{n-1} T(j)\right) + c*n \]
Multiply through by $n$:
\[n * T(n) = 2 * \left(\sum_{j=0}^{n-1} T(j)\right) + c*n^2 \]
$n$ is just a variable, so this must be true no matter what value we plug in for it. Try replacing $n$ by $n-1$.
\[(n-1) * T(n-1) = 2 * \left(\sum_{j=0}^{n-2} T(j)\right) + c*(n-1)^2 \]
So now both of these are true:
\[ \begin{eqnarray} n * T(n) & = & 2 * \left(\sum_{j=0}^{n-1} T(j)\right) + c*n^2 \\ (n-1) * T(n-1) & = & 2 * \left(\sum_{j=0}^{n-2} T(j)\right) + c*(n-1)^2 \end{eqnarray} \]
Subtract the 2nd equation from the first, obtaining: \[ nT(n)-(n-1)T(n-1)=2T(n-1)+2cn-c \]
Collect the $T(n-1)$ terms together and drop the $-c$ term: \[ n T(n)=(n+1)T(n-1)+2cn \]
Next we apply a standard technique for solving recurrence relations. Divide by $n(n+1)$ and “telescope”: \[ \begin{eqnarray*} \frac{T(n)}{n+1} & = & \frac{T(n-1)}{n} + \frac{2c}{n+1} \\ \frac{T(n-1)}{n} & = & \frac{T(n-2)}{n-1} + \frac{2c}{n} \\ \frac{T(n-2)}{n-1} & = & \frac{T(n-3)}{n-2} + \frac{2c}{n-1} \\ & \vdots & \end{eqnarray*} \]
Note that most of the terms on the left will have appeared on the right in the previous equation, so if we were to add up all these equations, these terms would appear on both sides and could be dropped: \[ \frac{T(n)}{n+1} = \frac{T(1)}{2} + 2 c \sum_{j=3}^{n+1} 1/j \]
As $n$ gets very large, $\sum_{j=3}^{n+1} 1/j$ approaches $\ln(n) + \gamma$, where $\gamma$ is Euler’s constant, $0.577\ldots$.
Hence \[ \begin{eqnarray*} \frac{T(n)}{n+1} & = & \frac{T(1)}{2} + 2 c * \ln(n) + 2 c \gamma \\ & = & \ln(n) + c_2 \\ & = & O(\ln(n)) \end{eqnarray*} \]
and so $T(n)$ is $O(n \log(n))$.