1 Divide and conquer

Perhaps the most widely recognized algorithmic style is “divide and conquer”. In this approach to design, a problem is broken into two or more smaller sub-problems, the sub-problems are solved, and the resulting sub-problem solutions are recombined into a whole solution.

Examples of divide and conquer that we have seen include the binary search, merge sort, and quick sort.

template <class InputIterator,
          class OutputIterator>
OutputIterator copy(InputIterator first, 
                    InputIterator last,
                    OutputIterator result)
{
  while (first != last)
    {
      *result = *first;
      result++; first++;
    }
  return result;
}

In fact, the STL (standard template library), that eventually became part of the C++ standard library, was originally constructed as a showcase for the idea of generic programming.

3 Generate and test

A “generate and test” algorithm uses some relatively quick process to produce a series of “guesses” as to the appropriate solution, then tests each guess in turn to see if it is, in fact, a solution.

You can see generate and test in action in this code for producing random permutations.

//
//  Generate a random permutation of the integers from
//  0 .. n-1, storing the results in array a.
//
void permute1 (int a[], int n)
{
  for (int i = 0; i < n; i++)
    {
      a[i] = rnd(n);
      while (find(a,a+i,a[i]) != a+i)
         a[i] = rnd(n);
    }
}

• The random numbers are used to generate a possible value for the $i^{\mbox{th}}$ number in the permutation.

• Then a search is used to test and see if that number has already been used.

Generate-and-test is often used as a fall-back when we can’t come up with a better algorithm.

4.1 Example: The 3-Houses Puzzle

An example of backtracking can be seen in the solution technique many people take to problems like this:

There are 3 adjacent houses, one red, one blue, and one green on Elm St. Each is occupied by a single person, and each has a garden with one kind of flower.

1. Bob does not live in the green house.

2. Pat lives between Bob and Sue.

3. Sue planted daisies.

4. Lilies are planted at the blue house.

5. The leftmost house has roses.

List the colors of the houses, the occupants, and their flowers, from left to right.

We use backtracking largely in cases where we can’t find a more sophisticated way to arrive at a solution.

4.2 A Tabular Approach to Backtracking

The essence of backtracking is:

1. Number the solution variables $[v_0, v_1, \ldots, v_{n-1}]$.

2. Number the possible values for each variable $[c_0, c_1, \ldots, c_{k-1}]$.

3. Start by assigning $c_0$ to each $v_i$.

4. If we have an acceptable solution, stop.

5. If the current solution is not acceptable, let i = n-1.

6. If i < 0, stop and signal that no solution is possible.

7. Let j be the index of the value assigned to variable $v_i$ (i.e., $v_i = c_j$). If $j < k-1$, assign $c_{j+1}$ to $v_i$ and go back to step 4.

8. But if $j \geq k-1$, assign $c_0$ to $v_i$, decrement i, and go back to step 6.

There are lots of variations possible. It’s fairly easy to modify this scheme to deal with situations where different variables have different sets of possible values.

Many game-playing programs use a form of this kind of backtracking to select the computer’s move.

4.2.1 Example: The 3-House Puzzle

There are 9 variables in this problem: color_L, color_M, color_R, flowers_L, flowers_M, flowers_R, person_L, person_M, and person_R, where the “L”, “M”, and “R” subscripts denote “left”, “Middle”, and “Right”, respectively.

Each can take on 3 possible values. Letting blue=daisies=Bob=0, green=lilies=Pat=1, and red=roses=Sue=2, we can represent our progress through the problem like this:

Start with none of the variables assigned.

colorL	colorM	colorR	flowersL	flowersM	flowersR	personL	personM	personR

Then assign red to the left house:

colorL	colorM	colorR	flowersL	flowersM	flowersR	personL	personM	personR
0

and green to the middle house:

colorL	colorM	colorR	flowersL	flowersM	flowersR	personL	personM	personR
0	1

Continuing on, we can show the whole progress (up to the point where we stopped earlier), showing each step as a separate line:

colorL	colorM	colorR	flowersL	flowersM	flowersR	personL	personM	personR
0
0	1
0	1	2
0	1	2	0
0	1	2	1
0	1	2	2
0	2
0	2	1

and so on.

Actually, even here we may have played fast and loose a little with the rules. How did we know that the middle house should start with color 1 instead of 0? Only because there is an implicit rule saying that there can only be a single house of each color. A violation of that rule is just another contradiction, so we could actually argue that the steps we went through were:

colorL	colorM	colorR	flowersL	flowersM	flowersR	personL	personM	personR
0
0	0
0	1
0	1	0
0	1	1
0	1	2
0	1	2	0
0	1	2	1
0	1	2	2
0	2
0	2	0
0	2	1

and so on. This is more typical of how we would be likely to program a solution.

1 of 6

4.3 A Recursive Approach to Backtracking

Here’s another way to think about this process.

This tree shows the “solution space” for a backtracking problem. At one level is the first problem variable with all of its possible solution values. Each node at the first level represents a possible value for the variable $v_0$.

The traditional way to program backtracking has been to write a recursive function to search this solution space. The first call deals with possible values for $v_0$, and loops through each one in turn. For each possible value of $v_0$, we recursively call the same routine, but instruct it to work with $v_1$ instead of $v_0$. That recursive call will loop through all possible values of $v_1$, trying each one and then recursively calling itself with instructions to work on $v_2$, and so on.

4.3.1 Example: Scheduling

The CS Dept. has a list of courses to offer next semester, and a list of available time slots in which to place them.

Some courses “conflict” – they cannot be scheduled at the same time because

• students may want to take both courses in the same semester

• the same faculty member is teaching both.

Suppose that the classes are numbered 0 ... NumClasses-1, and that the time slots are numbered 0 ... NumTimes-1.

We will assume that someone has already written a function

bool conflicts(int class1, int class2); 

to determine if two classes conflict.

vector<int> timeOfClass (numClasses, -1);

so that timeOfClass[c] is the time slot in which class c has been scheduled (-1 indicates that we have not yet assigned a time to c).

bool noConflicts (int class, int time) 
// Would assigning the given class to the given time
// cause a conflict with any already scheduled classes? 
// Return true if no conflicts would be caused.
{ 
  for (int c = 0; c < class; ++c) 
    {
     if (timeOfClass[c] = time   // if c is already schedule at this time
         && conflicts(c, class)) // and c conflicts with class
       return false;             // then we can't schedule class at this time
    }
  return true; 
} 

noConflicts simply examines every other class already in this time slot to see if it conflicts with this one.

bool schedule(int class) 
{ 
  if (class == NumClasses) 
    return true; 
  else 
    for (int t = 1; t <= NumTimes; ++t) 
      { 
       if (noConflicts(class, t)) 
         { 
          timeOfClass[class] = t; 
          if (schedule(i+1)) return true; 
         } 
      } 
  timeOfClass[class] = -1; 
  return false;    // all choices failed 
} 

• It does this by trying every plausible time slot for class i until it finds one that allows all the remaining classes to be (recursively) scheduled.

• If it can’t find a slot in which to place class i, it returns false.

  • Whenever a recursive call from schedule returns false back to an earlier call of itself, the earlier call is forced to try a different time slot for a previously scheduled class.

4.3.2 Worst-Case Analysis

bool noConflicts (int class, int time) 
// Would assigning the given class to the given time
// cause a conflict with any already scheduled classes? 
// Return true if no conflicts would be caused.
{ 
  for (int c = 0; c < class; ++c) 
    {
     if (timeOfClass[c] = time  
         && conflicts(c, class))
       return false;            
    }
  return true; 
} 

• Then each class will be tried in each time slot.
  • So for the first class, there will be NumTimes possibilities.
  • For each of those possibilities, the second class will have a further NumTimes possibilities of its own, for a total of $\mbox{NumTimes}^2$ possibilities for the two classes.
  • Continuing on, the total number of possibilities for all classes is $\mbox{NumTimes}^{\mbox{NumClasses}}$.

We say that this routine is exponential in NumClasses. Such exponential growth is worse, for sufficiently large inputs, than any polynomial big-O().

4.4 An Iterative Framework for Backtracking

Can we do this without the recursion?

Yes.

Let’s return again to the tree view of the possible solutions to a backtracking problem.

Let’s assume our tree has 5 levels, corresponding to $v_0$ through $v_4$, and that each variable can take on any of 3 values.

Now, suppose we list the possible solutions from left to right in this tree, listing each one from $v_{n-1}$ down to $v_0$.

• The leftmost path in the tree is 00000, then 00001, then 00002, 00010, 00011, 00012, 00020, … .
• You can see that all we are doing is counting in “base 3”. If our variables could have taken on 10 different values instead of just three, we’d be counting in our “traditional” base-10 number system.

So, if we had an ADT for counting N digit numbers in an arbitrary base, we could use that as a kind of generator for backtracking solutions.

4.4.1 The BackTrack class

The code shown here represents a backtracking state generator.

class BackTrack {
public:
  BackTrack (unsigned nVariables, unsigned arity=2);
  // Create a backtracking state for a problem with
  // nVariables variables, each of which has the same
  // number of possible values (arity).

  template <class Iterator>
  BackTrack (Iterator arityBegin,
             Iterator arityEnd);
  // Create a backtracking state in which each variable may have
  // a different number of possible values. The values are obtained
  // as integers stored in positions arityBegin .. arityEnd as per
  // the usual conventions for C++ iterators. The number of
  // variables in the system are inferred from the number of
  // positions in the given range.

  unsigned operator[] (unsigned variableNumber) const;
  // Returns the current value associated with the indicated
  // variable.

  unsigned numberOfVariables() const;
  // Returns the number of variables in the backtracking system.

  unsigned arity (unsigned variableNumber) const;
  // Returns the number of potential values that can be assigned
  // to the indicated variable.

  bool more() const;
  // Indicates whether additional candidate solutions exist that
  // can be reached by subsequent ++ or prune operations.

  void prune (unsigned level);
  // Indicates that the combination of values associated with
  // variables 0 .. level-1 (inclusive) has been judged unacceptable
  // (regardless of the values that could be given to variables
  // level..numberOfVariables()-1.  The backtracking state will advance
  // to the next solution in which at least one of the values in the
  // variables 0..level-1 will have changed.

  BackTrack& operator++();
  // Indicates that the combination of values associated with
  // variables 0 .. nVariables-1 (inclusive) has been judged unacceptable.
  // The backtracking state will advance
  // to the next solution in which at least one of the values in the
  // variables 0..level-1 will have changed.

  BackTrack operator++(int);
  // Same as other operator++, but returns a copy of the old backtrack state

private:
  bool done;
  vector<unsigned> arities;
  vector<unsigned> values;

};

BackTrack problem(9, 3);

would create backtracking state problem with 9 variables, each of which can take on the values 0, 1, or 2.

cout << "Variable " << i 
     << " has value " << problem[i]
     << endl;

BackTrack problem(4, 3);
// 4 questions, each with 3 possible answers
while (problem.more())
 {
   for (int i = 0; i < 4; ++i)
     cout << problem[i] << ' ';
   cout << endl;
   ++problem;
 }

the output would be: 0000, 0001, 0002, 0010, 0011, 0012, 0020, 0021, 0022, 0100, … , 2222.

BackTrack problem(nVariables, nValues); 
bool solved = false;
while ((!solved) && problem.more())
 {
   solved = checkSolution(problem);

   if (!solved)
     ++problem;
 }

where checkSolution is a function that returns true if the current problem state is an acceptable solution.

4.4.2 Example: Iterative Solution of 3-Houses

enum Questions {colorOfHouse1=0,
                colorOfHouse2=1, 
                colorOfHouse3=2, 
                flowersAtHouse1=3,
                flowersAtHouse2=4,
                flowersAtHouse3=5,
                occupantOfHouse1=6,
                occupantOfHouse2=7,
                occupantOfHouse3=8};

Although each group of three variables represents something different, it so happens that all the variables can take on three possible values.

int main()
{
  BackTrack problem(9, 3); 
    // 9 questions, each with 3 possible answers
  bool solved = false;
  while ((!solved) && problem.more())
    {
      for (int i = 0; i < 9; ++i)
        cout << problem[i] << ' ';
      cout << endl;

      solved = checkSolution(problem);
      
      if (!solved)
        ++problem;
    }

  if (solved)
    {
      describeHouse(1,
                    (Occupants)problem[occupantOfHouse1],
                    (Colors)problem[colorOfHouse1], 
                    (Flowers)problem[flowersAtHouse1]);
      describeHouse(2,
                    (Occupants)problem[occupantOfHouse2],
                    (Colors)problem[colorOfHouse2], 
                    (Flowers)problem[flowersAtHouse2]);
      describeHouse(3,
                    (Occupants)problem[occupantOfHouse3],
                    (Colors)problem[colorOfHouse3], 
                    (Flowers)problem[flowersAtHouse3]);
    }
  else
    cout << "Problem has no solution" << endl;
}

We’ll take this in stages.

The first rules to be checked are implicit ones. Bob can’t live in more than one house in this problem, and neither can Sue or Pat. So if any two houses have the same occupant, set OK to false. Note that bt[occupantOfHouse1] denotes the current assignment, in bt, to the occupantOfHouse1 variable. It will either be bob (0), pat (1), or sue (2).

int indexOf (const BackTrack& bt, int value,
             int candidate1, int candidate2, 
             int candidate3)
{
  if (bt[candidate1] == value)
    return candidate1;
  else if (bt[candidate2] == value)
    return candidate2;
  else
    return candidate3;
}

0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 1 
0 0 0 0 0 0 0 0 2 
0 0 0 0 0 0 0 1 0 
0 0 0 0 0 0 0 1 1 
0 0 0 0 0 0 0 1 2 
0 0 0 0 0 0 0 2 0 
0 0 0 0 0 0 0 2 1
0 0 0 0 0 0 0 2 2
0 0 0 0 0 0 1 0 0
    ⋮

… and, about 4000 lines later …

    ⋮
0 1 2 2 1 0 0 0 2 
0 1 2 2 1 0 0 1 0 
0 1 2 2 1 0 0 1 1 
0 1 2 2 1 0 0 1 2 
Bob lives in house 1, which is painted red and has roses
Pat lives in house 2, which is painted blue and has lilies
Sue lives in house 3, which is painted green and has daisies

One of the most common improvements is to use an some kind of “early” test to determine if a particular assignment to a particular variable is plausible rather than always waiting until all variables have been assigned to do any checking.

• For example, in the 3-houses puzzle, we can immediately reject any proposed solution that has Pat in houses 1 or 3, and there’s no point cycling through all the combinations of the other variables once we see this.

Note that the effect of “pruning” is most significant when it can occur early in the tree. A lot of research and design effort has gone into ideas like trying to reorder the variables and/or values so that pruning will occur as early as possible.

4.5.1 Pruning and the BackTrack class

Our backtracking state generator can easily accommodate pruning.

class BackTrack {
public:
  ⋮
  void prune (unsigned level);
  // Indicates that the combination of values associated with
  // variables 0 .. level-1 (inclusive) has been judged unacceptable
  // (regardless of the values that could be given to variables
  // level..numberOfVariables()-1.  The backtracking state will 
  // advance to the next solution in which at least one of the 
  // values in the variables 0..level-1 will have changed.
  ⋮

In fact, the “normal” ++ operator is actually implemented as prune(numberOfVariables()) (which is no pruning at all.)

void BackTrack::prune (unsigned level)
  // Indicates that the combination of values associated with
  // variables 0 .. level-1 (inclusive) has been judged unacceptable
  // (regardless of the values that could be given to variables
  // level..numberOfVariables()-1.  The backtracking state will advance
  // to the next solution in which at least one of the values in the
  // variables 0..level-1 will have changed.
{
  level = (level > numberOfVariables()) ? numberOfVariables() : level;
  fill (values.begin()+level, values.end(), 0);
 

  // Treat the top level-1 values as a level-1 digit number. Add one
  // to the rightmost "digit". If this digit goes too high, reset it to
  // zero and "carry one to the left".
  int k = level-1;
  bool carry = true;
  while (k >= 0 && carry)
    {
      values[k] += 1;
      if (values[k] >= arities[k])
        values[k] = 0;
      else
        carry = false;
      --k;
    }
  done = carry;
}

The variables being “pruned”, the ones in positions level … numberOfVariables(), are simply set to zero.

BackTrack& BackTrack::operator++()
  // Indicates that the combination of values associated with
  // variables 0 .. nVariables-1 (inclusive) has been judged unacceptable.
  // The backtracking state will advance
  // to the next solution in which at least one of the values in the
  // variables 0..level-1 will have changed.
{
  prune(numberOfVariables());
  return *this;
}

Applying Pruning

Now, to actually use pruning in a problem requires a bit more effort when we check the solution state. Instead of simply indicating whether the state is acceptable, we now want to indicate the smallest variable number that we want to change. Basically, for any rule that involves variables $v_j$ and $v_k$, if we find that the rule is violated, then the larger of j or k will need to be changed next.

4.6 Example: Pruning in 3-Houses

Instead of a simple boolean OK variable, we now use an integer to track the variable that we want changed on the next step.

int checkSolution(const BackTrack& bt)
  // Check the state of bt to see if it represents a valid solution.
  // Return -1 if a solution. If not a solution, return the smallest
  // number of any problem variable known to be incorrect.
{
  int knownIncorrect = bt.numberOfVariables();

  // Implicit rule: each house is different
  if (bt[occupantOfHouse1] == bt[occupantOfHouse2])
    knownIncorrect = min(knownIncorrect, 
                       max(occupantOfHouse1, occupantOfHouse2));
    ⋮

If we fail the test, we replace that integer with the larger of the variable numbers involved in the test, provided that we haven’t already found a lower-numbered variable for pruning.

⋮
  if (bt[occupantOfHouse1] == bt[occupantOfHouse3])
    knownIncorrect = min(knownIncorrect,
                         max(occupantOfHouse1, 
                             occupantOfHouse3));
  if (bt[occupantOfHouse2] == bt[occupantOfHouse3])
    knownIncorrect = min(knownIncorrect,
                         max(occupantOfHouse2, 
                             occupantOfHouse3));
    ⋮

⋮
  // 1. Bob does not live in the green house.
  int bobsHouse = indexOf(bt, bob,
                          occupantOfHouse1,
                          occupantOfHouse2,
                          occupantOfHouse3);
  int greenHouse = indexOf(bt, green,
                          colorOfHouse1,
                          colorOfHouse2,
                           colorOfHouse3);

  if (bobsHouse - occupantOfHouse1 == greenHouse - colorOfHouse1)
    knownIncorrect = min(knownIncorrect,
                         max(bobsHouse, greenHouse));
    

  // 2. Pat lives between Bob and Sue.
  if (bt[occupantOfHouse2] != pat)
    knownIncorrect = min(knownIncorrect, occupantOfHouse2);
    ⋮

⋮
  if (knownIncorrect >= bt.numberOfVariables())
    return -1;
  else
    return knownIncorrect;
}

4.6.1 Output After Pruning

If we run this version, the output is

0 0 0 0 0 0 0 0 0 
0 1 0 0 0 0 0 0 0 
0 1 1 0 0 0 0 0 0 
0 1 2 0 0 0 0 0 0 
0 1 2 1 0 0 0 0 0 
0 1 2 2 0 0 0 0 0 
0 1 2 2 0 1 0 0 0 
0 1 2 2 0 2 0 0 0 
0 1 2 2 1 0 0 0 0 
0 1 2 2 1 0 0 1 0 
0 1 2 2 1 0 0 1 1 
0 1 2 2 1 0 0 1 2 
Bob lives in house 1, which is painted red and has roses
Pat lives in house 2, which is painted blue and has lilies
Sue lives in house 3, which is painted green and has daisies

That’s the whole thing! Only 12 potential solutions were examined, instead of more than 4000.

This shows that pruning can be critical to making backtracking work efficiently. But don’t get carried away. The amount of reduction you get from pruning varies considerably (we were probably a bit lucky here) and, even with pruning, the worst case is still exponential time.

Many scientific and engineering programs depend upon convergent algorithms.

6 Dynamic programming

Dynamic programming is a replacement for recursion that is useful when a recursive solution would repeatedly solve the same small subproblems.

The basic idea is to go ahead and solve the smaller problems, but to save the results in a data structure so that, if you need the same subproblem solved again, you can just fetch the prior answer from the data structure.

6.1 Example: combinations

Suppose we have a set of $n$ distinct items, and need to select $k$ of them at random (e.g., dealing 5 cards from a deck of 52). Assume that we aren’t concerned about the order in which we obtained the selected items, but we want to know how many different possible sets of items we could possible obtain.

The number of different combinations we can obtain by selecting k items from a set of n, without replacement, is written as $\left(\begin{array}{c} n \\ k \end{array} \right)$ or sometimes as $C(n,k)$.

It can be evaluated as

\[ C(n,n) = C(n,0) = 1 \]

\[ C(n,k) = C(n - 1, k - 1) + C(n - 1, k), \; 0 < k < n \]

long C(int n, int k) 
{ 
  if (k==0 || k==n) 
    return 1; 
  else 
    return C(n-1,k-1) + C(n-1,k); 
} 

But this wastes effort re-evaluating smaller problem over and over again. For example,

\[ C(5,2) = C(4,1) + C(4,2) \]

No problem so far, but let’s expand each of the items on the right

\[ \begin{eqnarray*} C(4,1) & = & C(3,0) + C(3,1) \\ C(4,2) & = & C(3,1) + C(3,2) \end{eqnarray*} \]

and you can start to see that one subproblem $C(3,1)$ has already arisen in two different places.

Furthermore, if we were to expand these again, we would find that both occurrences of $C(3,1)$ and $C(3,2)$ will need the value of $C(2,1)$. In fact, most of the run time of this algorithm will be devoted to solving and re-solving the same smaller problems, over and over.

6.2 From Recursion to Dynamic Programming

Reversing the Calculation

But wait — there’s more!

Those recursive calls, when they take place, always ask for the solution to a “smaller” problem (either n, or k, or both are smaller).

• That’s typical of recursion – the recursive calls always need to be approaching some base case.
• Otherwise the recursive calculation might not terminate.

But in this problem, there are only so many possible smaller problems.

Because we never need the answer to a larger problem before we can answer a smaller one, we can turn the whole evaluation inside-out and simply start by evaluating the simple problems, then the larger problems that can be solved from those, and so on, until we reach the original problem that we were really interested in.

Dynamic programming frequently not only prevents repeated recursion over common sub-problems, but often introduces a structure over which a purely iterative solution is possible.

6.3 Example: Edit Distance

One of the classic problems solvable by dynamic programming is computing the “edit distance” between two strings: the minimum number of single-character insertions, deletions, or replacements required to convert one string into another.

• For example, the edit distance between “Hello” and “Jello” is 1.
• The edit distance between “good”
  and “goodbye” is 3.
• The edit distance between any string and itself is 0.

The edit distance can be used for such purposes as suggesting, in a spell checker, a list of plausible replacements for a misspelled word. For each word not found in the dictionary (and therefore presumably misspelled), list all words in the dictionary that are a small edit distance way from the misspelling.

6.3.1 Breaking the Problem Down

We can compute the edit distance between two strings by working from the ends. The three operations available to us are

• Add one character to the end of a string

• Remove one character from the end of a string

• Change the character at the end of a string.

• If we knew how to convert “Zeil” to “trial”, we could add “s” to get the desired word.

• If we knew how to convert “Zei” to “trials”, then we would actually have “trialsl” and we could remove that last character to get the desired word.

• If we knew how to convert “Zei” to “trial”, then we would actually have “triall” and we could change the final “l” to “s” to get the desired word.

6.3.2 Breaking it Down - Stage 2

We continue, recursively, to break down these problems:

• Convert “Zeil” to “trial”, then add “s” to get the desired word.

  To convert “Zeil” to “trial”,

  Add

  Convert “Zeil” to “tria”, then add “l”

  Remove

  Convert “Zei” to “trial”, giving “triall”, then remove.

  Change(?)

  Convert “Zei” to “tria”, giving “trial”, and no change is actually needed.

• Convert “Zei” to “trials”, giving “trialsl”, then remove the last character.

  To convert “Zei” to “trials”,

  Add

  Convert “Zei” to “trial”, then add “s”

  Remove

  Convert “Ze” to “trials”, giving “trialsi”, then remove.

  Change

  Convert “Ze” to “trial”, giving “triali”, and change the final character.

• Convert “Zei” to “trial”, giving “triall”, then change the final “l” to “s”.

  To convert “Zei” to “trial”,

  Add

  Convert “Zei” to “tria”, then add “l”

  Remove

  Convert “Ze” to “trial”, giving “triali”, then remove.

  Change

  Convert “Ze” to “tria”, giving “triai”, and change the final character.

Convert "" to "xyz",' 

which can be trivially solved by a series of “Adds”.

6.3.3 A Recursive Solution

Here you see the recursive implementation of the edit distance calculation.

int editDistance (string x, string y)
{
  if (x == "")
     return y.length(); // base case
  else if (y == "")
     return x.length(); // base case
  else
    {
      int addDistance = editDistance(x, y.substr(0,y.length()-1)) + 1;
      int removeDistance = editDistance(x.substr(0,x.length()-1), y) + 1;
      int changeDistance = editDistance(x.substr(0,x.length()-1), 
                                        y.substr(0,y.length()-1)) 
        + (x[x.length()-1] == y[y.length()-1]) ? 0 : 1;
      return min(min(addDistance, removeDistance), changeDistance);
    }
}

In the main portion, we don’t know, off hand, whether the cheapest way to convert one string into another involves a final add, remove, or change, so we evaluate all three possibilities and return the minimum distance from among the three.

In each case, we recursively compute the distance (number of adds, removes, and changes) required to “set up” a final add, a final remove, or a final change. We add one to the add distance and the remove distance to account for the final add or remove. For the change distance, we add one only if the final characters in the strings are different (if not, no final change is required).

6.3.4 A Dynamic Programming Solution

We can solve the same problem via dynamic programming by, again, reversing the direction so that we work the smaller subproblems first, keeping the answers in a table.

For example, in converting “Zeil” to “trials”, we start by forming a table of the cost (edit distance) to convert "“ to ”“, ”t“, ”tr“, ”tri", etc.:

(The symbol λ denotes the empty string.)

In other words, we need 0 steps to convert "“ to ”“, 1 to convert ”“ to ”t“, 2 to convert ”“ to ”tr", and so on.

OK, clearly we need 1 step to convert “Z” to "". How are the other entries in this row computed?

Let’s back up just a bit:

What’s the minimum cost to convert “Z” to “t”? It’s the smallest of the three values computed as

Add

1 plus the cost of converting “Z” to "" (we get this cost by looking in the table to the left one position).

Remove

1 plus the cost of converting "“ to ”t“, giving ”tZ" (we get this cost by looking up one position).

Change

1 (because “Z” and “t” are different characters) plus the cost of converting "“ to ”" (we get this cost by looking diagonally up and to the left one position).

The last of these yields the minimal distance: 1.

What’s the minimum cost to convert “Z” to “tr”? It’s the smallest of the three values computed as

Add

1 plus the cost of converting “Z” to “t” (we get this cost by looking to the left one position).

Remove

1 plus the cost of converting "“ to ”tr“, giving ”trZ" (we get this cost by looking up one position).

Change

1 (because “Z” and “t” are different characters) plus the cost of converting "“ to ”t" (we get this cost by looking diagonally up and to the left one position).

The last of these yields the minimal distance: 2.

Filling In The Table

And we add the next row, using the same technique:

… we are looking at the cost of converting “Zei” to “tri”. It’s the smallest of the three values computed as

Add

1 plus the cost of converting “Zei” to “tr” (we get this cost by looking to the left one position).

Remove

1 plus the cost of converting “Ze” to “tri”, giving “trii” (we get this cost by looking up one position).

Change

Zero (because “i” and “i” are the same character) plus the cost of converting “Ze” to “tr” (we get this cost by looking diagonally up and to the left one position).

The last of these yields the minimal cost of 2.

Note that this last row, again, has a situation where the cost of a change is zero plus the subproblem cost, because the two characters involved are the same (“l”).

From the lower right hand corner, then, we read out the edit distance between “Zeil” and “trials” as 4.

int editDistance (const string& x, const string& y)
{
  Matrix<int> distances (x.size()+1, y.size()+1);

  // Initialize the matrix
  for (int ix = 0; ix <= x.size(); ++ix)
    distances[ix][0] = ix;
  for (int iy = 0; iy <= y.size(); ++iy)
    distances[0][iy] = iy;

  // Fill in rest of the matrix from left to right and top to bottom
  for (int iy = 1; iy <= y.size(); ++iy)
    for (int ix = 1; ix <= x.size(); ++ix)
      {
        int costByDeleteOrInsert = min(1+distances[ix-1][iy],
                                       1+distances[ix][iy-1]);

        int costByReplacement = distances[ix-1][iy-1] +
          ((x[ix-1] == y[iy-1]) ? 0 : 1);

        distances[ix][iy] = min(costByDeleteOrInsert, costByReplacement);
      }
  

  return distances[x.size()][y.size()];
}

It’s pretty clear that this algorithm is O(x.size()*y.size()), both in time and in memory use.

int editDistance (const string& x, const string& y)
{
   int* priorColumn = new int[x.size()+1];
   int* currentColumn = new int[x.size()+1];

  // initialize column 0
  for (int ix = 0; ix <= x.size(); ++ix)
    priorColumn[ix] = ix;

  // Fill in rest of the matrix from left to right and top to bottom
  for (int iy = 1; iy <= y.size(); ++iy)
    {
	  currentColumn[0] = iy;
      for (int ix = 1; ix <= x.size(); ++ix)
        {
         int costByDeleteOrInsert = min(1+currentColumn[ix-1],
                                        1+priorColumn[ix]);

        int costByReplacement = priorColumn[ix-1] +
          ((x[ix-1] == y[iy-1]) ? 0 : 1);

        currentColumn[ix] = min(costByDeleteOrInsert, costByReplacement)a;

      }
      swap (priorColumn, currentColumn);
    }
  
  int distance = priorColumn[x.size()];
  delete [] priorColumn;
  delete [] currentColumn;
  return distance;
}

This algorithm is still O(x.size()*y.size()) in time but only O(x.size()) in memory use.

6.4 Designing Dynamic Programming Solutions

1. Start with a recursive solution.

2. Examine the parameters to the recursive function, e.g., f(x, y, z).

   • How many of these parameters change from one recursive call to another?

   • Are they all integers?

   • Do they all decrease/stay the same in each recursive call?

If the changing paramters consist of $k$ non-increasing integer parameters,…

• A dynamic program will need a table of dimension $k$.
• The base cases of the recursion are used to initialize the lowest-numbered rows & columns in the table.
• The calculation of each new table cell replaces a recursive call $f(x-c_1, y-c_2, \ldots)$ by $t[x-c_1, y-c_2, \ldots]$.

6.4.1 Example: Combinations

long C(int n, int k) 
{ 
  if (k==0 || k==n) 
    return 1; 
  else 
    return C(n-1,k-1) + C(n-1,k); 
} 

There are two parameters, they both change, so

• A dynamic program will need a table of dimension $2$. E.g.,

  int** t = t[n+1][k+1];
  

• We can initialize the table t[n,k] for all values t[n,0] and t[i,i]

• The calculation of each new table cell replaces a recursive call C(n-1,k-1) by t[n-1,k-1] and C(n-1,k) by t[n-1,k], e.g.,

  t[n,k] = t[n-1,k-1] + t[n-1,k];
  

6.4.2 Example: Edit Distance

int editDistance (string x, string y)
{
  if (x == "")
     return y.length(); // base case
  else if (y == "")
     return x.length(); // base case
  else
    {
      int addDistance = editDistance(x, y.substr(0,y.length()-1)) + 1;
      int removeDistance = editDistance(x.substr(0,x.length()-1), y) + 1;
      int changeDistance = editDistance(x.substr(0,x.length()-1), 
                                        y.substr(0,y.length()-1)) 
        + (x[x.length()-1] == y[y.length()-1]) ? 0 : 1;
      return min(min(addDistance, removeDistance), changeDistance);
    }
}

Althugh this function uses strings instead of integers, we can “convert” it to integers by looking at the length of the substrings being examined:

int editDistance (string x, string y, int xLen, int yLen)
{
  if (xLen == 0)
     return yLen; // base case
  else if (yLen == 0)
     return xLen; // base case
  else
    {
      int addDistance = editDistance(x, y, x.substr(0,xLen), y.substr(0,yLen-1)) + 1;
      int removeDistance = editDistance(x, y, x.substr(0,xLen), y.substr(0,yLen)) + 1;
      int changeDistance = editDistance(x, y, x.substr(0,xlen-1), 
                                        y.substr(0,yLen-1)) 
        + (x[xLen-1] == y[yLen-1]) ? 0 : 1;
      return min(min(addDistance, removeDistance), changeDistance);
    }
}

Then we can observed that, of the 4 parameters in the call, only 2 are changing.

• A dynamic program will need a table of dimension $2$. E.g.,

  int** t = t[x.size()+1][y.size()+1];
  

• We can initialize the table t[xLen,yLen] for all values t[xLen,0] and t[0,yLen]

• The calculation of each new table cell replaces a recursive call editDistance(x,y,i,j) by t[i,j] e.g.,

  int addDistance = t[i,j-1] + 1;
  int removeDistance = t[i,j] + 1;
  int changeDistance = t[i-1, j-1] + (x[i-1] == y[j-1]) ? 0 : 1;
  t[i,j] = min(min(addDistance, removeDistance), changeDistance);