Heaps

Steven J. Zeil

Last modified: Nov 20, 2023
Contents:
1 Recap: the std Priority Queue Interface
1.1 The Priority Queue Implementation
2 Implementing Priority Queues - the Heap
2.1 Binary Heaps
2.2 Heaps are complete trees
2.3 Children’s Values are Smaller than Their Parent’s
2.4 Bubbling and Percolating
2.5 Inserting into a heap
2.6 Removing from Heaps
3 Analysis
3.1 Building a Heap
4 Implementing the Heap with Iterators
4.1 push_heap
4.2 pop_heap
4.3 percolateDown
4.4 make_heap
5 Recap: Complexity of Heap Operations

Problem: Given a collection of elements that carry a numeric “score”, find and remove the element with the smallest [largest] score. New elements may be added at any time.

In an earlier lesson, we saw that this collection is called a priority queue. Now we will look at an efficient way of implementing it.

1 Recap: the std Priority Queue Interface

This is the entire priority queue interface. 

template <class T, class Container, class Compare> 
class  priority_queue 
{
public:
  typedef T value_type;
  typedef Container::size_type size_type;

protected:
  Container c;
  Compare comp;

public:
  priority_queue(const Compare& x = Compare());

  template <class Iterator>
  priority_queue(Iterator first,
                 Iterator last, 
                 const Compare& x = Compare());

  bool empty() const;

  size_type size() const;

  value_type& top();

  const value_type& top() const;

  void push(const value_type& x);

  void pop();
};

1.1 The Priority Queue Implementation

template <class T, class Container, class Compare> 
class  priority_queue 
{
public:
  typedef T value_type;
  typedef Container::size_type size_type;

protected:
  Container c;
  Compare comp;

public:
  priority_queue(const Compare& x = Compare()) 
    :  c(), comp(x) {}

  template <class Iterator>
  priority_queue(Iterator first,
                 Iterator last, 
                 const Compare& x = Compare()) 
    : c(first, last), comp(x) 
    {
      make_heap(c.begin(), c.end(), comp); ➀
    }

  bool empty() const { return c.empty(); }

  size_type size() const { return c.size(); }

  value_type& top() { return c.front(); }

  const value_type& top() const 
    { return c.front(); }

  void push(const value_type& x) 
    { 
      c.push_back(x); 
      push_heap(c.begin(), c.end(), comp); ➁
    }

  void pop() 
    { 
      pop_heap(c.begin(), c.end(), comp); ➂
      c.pop_back(); 
    }
};

Most of the work is in the 3 algorithm fragments, make_heap ➀ , push_heap ➁ , and pop_heap ➂ . These are template functions from the standard library that are used to implement a “heap”, a new data structure that we will look at in detail shortly.

First, though, let’s look at what we could do to implement priority queues using the data structures we already know.

One possibility would be to use a sorted sequential structure (array, vector, or list). For example, using a vector, we would try to keep the elements in ascending order by priority. Then we could get the top() of the priority queue as the back() of the implementing vector.

Question: With this data structure, what would the complexities of the priority queue push and pop operations be?

Answer

We can do better than that.

We might consider instead using a balanced binary search tree to store the priority queue. This time, it will be a little easier if we store the items in descending order by priority. We could get the top() of the priority queue as *begin().

Question:

Using a balanced binary search tree as the underlying data structure, what would the complexities of the priority queue push and pop operations be?

Answer

That sounds pretty good.

We can’t actually hope to improve on the worst case times offered by balanced search trees, we can match those worst case times (and improve on the multiplicative constant) and actually achieve O(1) average case complexities by the use of a new data structure, called a “heap”.

2 Implementing Priority Queues - the Heap

We can implement priority queues using a data structure called a (binary) heap.

2.1 Binary Heaps

A binary heap is a binary tree with the properties:

**Important: A heap is a binary tree, but not a binary search tree. The ordering rules for heaps are different from those of binary search trees.

What I have defined here is sometimes called a max-heap, because the largest value in the heap will be at the root. We can also have a min-heap, in which every child has a value larger than its parent.

Max-heaps always have their very largest value in their root. Min-heaps always have their smallest value in the root.

In this course, we will always assume that a “heap” is a “max-heap” unless explicitly stated otherwise.

Let’s look at the implications of each of these two properties.

2.2 Heaps are complete trees

 

Here’s an example of a complete binary tree.

Complete binary trees have a very simple linear representation, allowing us to implement them in an array or vector with no pointers.

 

2.3 Children’s Values are Smaller than Their Parent’s

 

Using the same tree shape, we can fill in some values to show an example of a heap.

Each parent has a value larger than its children’s values (and, therefore, larger than the values of any of its descendants).

So when we ask for the front (largest value) of a priority queue, we find it in the root of the heap, which in turn will be in position 0 of the array/vector.

2.4 Bubbling and Percolating

Before looking in detail at how to add and delete elements from a heap, let’s consider a situation in which we have a “damaged” heap with one node out of position.

How do we “fix” the heap? There are two cases to consider.

2.4.1 Bubble Up

 

When we have a node that is larger than its parent, we bubble it up by swapping it with its parent until it has reached its proper position. 

void bubbleUp (vector<T>& heap, 
               unsigned nodeToBubble)
{
  unsigned parent = (nodeToBubble - 1) / 2;
  while (node > 0 && heap[nodeToBubble] > heap[parent])
    {
     swap(heap[nodeToBubble], heap[parent]);
     nodeToBubble = parent;
     parent = (nodeToBubble - 1) / 2;
    }
}

 

In this case, starting with nodeToBubble = 8, we swap node 8 with its parent 3 …

first prev1 of 3next last

Note that we have repaired the heap. The final arrangement satisfies the ordering requirements for a heap.

2.4.2 Percolate Down

 

When we have a node that is smaller than one or both of its children, we percolate it down by swapping it with the larger of its children until it has reached its proper position.

void percolateDown (vector<T>& heap, 
                    unsigned nodeToPerc)
{
  while (2*nodeToPerc+1 < heap.size())
    {
      unsigned child1 = 2*nodeToPerc +1;
      unsigned child2 = child1+1;

      unsigned largerChild = child1;
      if (child2 < heap.size() 
          && heap[child2] > heap[child1])
        largerChild = child2;
      if (heap[largerChild] > heap[nodeToPerc])
        {
          swap (heap[nodeToPerc], heap[largerChild]);
          nodeToPerc = largerChild;
        }
      else
        nodeToPerc = heap.size();
    }
}

This is only a little more complicated than bubbling up. The main complication is that the current node might have 0 children, 1 child, or 2 children, so we need to be careful that we don’t try to access the value of non-existent children.

 

In this case, starting with nodeToPerc = 0, we swap node 0 with its larger child, 2 …

first prev1 of 3next last

2.5 Inserting into a heap

 

Suppose we have this heap and we want to add a new item to it.

Now, after we add an item to the heap, it will have one more tree node than it currently does. Because heaps are complete trees, we know exactly how the shape of the tree will change, even if we can’t be sure how the data values in the tree might be rearranged.

 

Question: How will the shape of the tree shown above change?

Answer:

 

Well, suppose that we just go ahead and put the new value into that position.

We’ve got two possibilities.

It would be the only node that was out of position, and we know how to “repair” a heap with a single node out of position that is larger than its parent — we bubble up!

void add_to_heap (vector<T>& heap, const T& newValue)
{
  heap.push_back (newValue); // add newValue to complete tree
  bubbleUp (heap, heap.size()-1); // repair the heap
}

 

For example, suppose we wanted to add 54 to the heap. First we would push 54 onto the end of the vector, in effect adding it to the complete tree.

first prev1 of 3next last

2.6 Removing from Heaps

When we pop, or remove, from a heap, we know that the value being removed is the value currently in the root.

We also know how the tree shape will change. The rightmost node in the bottom level will disappear.

Now, unless the heap only has one node, the node that’s disappearing does not contain the value that we’re actually removing. So, we have two problems:

So, we’ve got a node with no data, and data that needs a node. The natural thing to do is to put the data in that node.

That data value will almost certainly be out of position, being smaller than one or both of its children, but, again, that’s only a single node that’s out of position. We know how to fix that.

void remove_from_heap (vector<T>& heap)
{
  heap[0] = heap[heap.size()-1]; // replace root value
  heap.pop_back(); // remove the duplicate node
  percolateDown (heap, 0); // repair the heap
}

 

Suppose we wanted to remove the maximum value from this heap.

The first step is to replace the root value by 47.

Then we pop the back of the vector to remove that final node.

first prev1 of 4next last

3 Analysis

A binary heap has the same shape as a balanced binary search tree.

Therefore its height, for $n$ nodes, is $\left\lfloor \log(n) \right\rfloor$.

push_heap and pop do $O(1)$ work on each node along a path that runs, at worst, between a single leaf and the root.

Hence both operations are $O(\log n)$, worst case.

The average case for push_heap is $O(1)$. The proof of this is beyond scope of this class.

3.1 Building a Heap

A single insertion is $O(\log n)$ worst case and O(1) average.

What happens if we start with an empty heap and do $n$ inserts? The resulting total could be $O(n \log n)$.

As it happens, we can do better with a special make_heap operation to build an entire heap from an array (or array-like structure such as a vector).

void build_heap (vector<T>& heap)
{
  unsigned i = (heap.size()-1)/2;
  do {
    percolateDown (heap, i);
    --i;
  } while (i >= 0);
}

  • Start with the data in any order.

  • Force heap order by percolating each non-leaf node.

Since each percolateDown takes, in worst case, a time proportional to the height of the node being percolated, the total time for build_heap is proportional to the sum of the heights of all the nodes in a complete tree. It is possible to show that this sum is itself $O(n)$, where $n$ is the number of nodes in the tree. Therefore build_heap is $O(n)$.

So it’s cheaper to build a heap all at once than to do it one push at a time, although neither approach is terribly expensive.

4 Implementing the Heap with Iterators

I’ve shown the operations add_to_heap, remove_from_heap, and build_heap in a simplified form so far. I’ve used [ ] indexing to get access to the elements because doing so makes the child/parent position calculations easier to read.

But the actual std function templates for heap manipulation use iterators, so let’s look at the “real thing” now.

4.1 push_heap

template <class RandomIterator, class Compare>
void push_heap (RandomIterator first, 
                RandomIterator last,
                Compare comp)
//Pre: The range [first,last-1) is a valid heap.
//Post: Places the value in the location last-1 into the resulting
//      heap [first, last).

{
  RandomIterator nodeToBubble = last - 1;
  RandomIterator parent = first + (nodeToBubble - first - 1)/2;
  while (nodeToBubble != first && comp(*parent, *nodeToBubble))
     {
      swap(*nodeToBubble, *parent);
      nodeToBubble = parent;
      parent = first + (nodeToBubble - first - 1)/2;
     }
}

void bubbleUp (vector<T>& heap, 
               unsigned nodeToBubble)
{
  unsigned parent = (nodeToBubble - 1) / 2;
  while (node > 0 && heap[nodeToBubble] > heap[parent])
    {
     swap(heap[nodeToBubble], heap[parent]);
     nodeToBubble = parent;
     parent = (nodeToBubble - 1) / 2;
    }
}

Compare that code for push_heap to our earlier code for bubbleUp. You can see that push_heap is pretty much a 1-to-1 translation of bubbleUp in which we have made the following changes:

  • Replaced unsigned indices by iterators

  • Replaced indexing [ ] by the iterator *

  • The parent calculations are offset by the starting position first

  • The comparison function (or functor) comp is used to compare items instead of <.

4.2 pop_heap

The std::pop_heap function relies on a utility, _percolateDown. [^ The “_” on the front of the function name _percolateDown is a signal that this function name is not part of the C++ standard, and so may not actually exist in your compiler’s standard library, unlike pop_heap or push_heap, which are part of the standard.]

template <class RandomIterator, class Compare>
void pop_heap (RandomIterator first, 
               RandomIterator last,
               Compare comp)
//Pre: The range [first,last) is a valid heap.
//Post: Swaps the value in location first with the value in the location
//      last-1 and makes [first, last-1) into a heap.
{
  swap (*first, *(last-1));
  _percolateDown (first, last-1, comp, 0, last-first-1);
}

4.3 percolateDown

The utility function _percolateDown … 

template <class RandomIterator, class Compare, class Distance>
void _percolateDown 
    (RandomIterator first, 
     RandomIterator last,
     Compare comp,
     Distance nodeToPerc,
     Distance heapSize)
{
  while (2*nodeToPerc+1 < heapSize)
    { 
      Distance child1 = 2*nodeToPerc +1;
      Distance child2 = child1+1;
      Distance largerChild = child1;
      if (child2 < heapSize 
          && *(first + child2) > *(first+child1))
        largerChild = child2;
      if (*(first + largerChild) > *(first + nodeToPerc))
        {
          swap (*(first + nodeToPerc), *(first + largerChild));
          nodeToPerc = largerChild;
        }
      else
        nodeToPerc = heapSize;
    }
}

… differs from our earlier percolateDown … 

void percolateDown (vector<T>& heap, 
                    unsigned nodeToPerc)
{
  while (2*nodeToPerc+1 < heap.size())
    {
      unsigned child1 = 2*nodeToPerc +1;
      unsigned child2 = child1+1;

      unsigned largerChild = child1;
      if (child2 < heap.size() 
          && heap[child2] > heap[child1])
        largerChild = child2;
      if (heap[largerChild] > heap[nodeToPerc])
        {
          swap (heap[nodeToPerc], heap[largerChild]);
          nodeToPerc = largerChild;
        }
      else
        nodeToPerc = heap.size();
    }
}

… by the same changes as we saw for push_heap and bubbleUp.

4.4 make_heap

make_heap uses the same _percolateDown utility as does pop_heap. 

template <class RandomIterator, class Compare>
void make_heap (RandomIterator first, 
                RandomIterator last,
                Compare comp)
//Pre: 
//Post: Arranges the values in [first, last) into a heap.
{
  if (first != last)
    {
      RandomIterator i = first+(last-first-1)/2 + 1; 
      while (i != first) 
        {
          --i;
          _percolateDown (first, last, comp, i-first, last-first);
        }
    }
}

It invokes _percolateDown on each non-leaf node, working back towards the root.

Try out the heap algorithms in an animation.

4.4.1 Complexity

make_heap is $O(N)$ worst case.

 

Consider an array with $N$ elements. Let $h$ be the height of the complete tree repsenting that array. $h = \log N$

We apply percolate to the first $N/2$ elements.

Total work: $\sum_{i=0}^h i \frac{N}{2^{i-1}}$

\[ = \sum_{i=0}^h N \frac{i}{2^{i-1}}\]

\[ = N * \sum_{i=0}^h \frac{i}{2^{i-1}} \]

Using one of our simplifications from the FAQ,

\[ < N * 3 \]

\[ = O(N) \]

5 Recap: Complexity of Heap Operations

© 2016-2023, Old Dominion Univ.