Testing ADTs in C++

Steven J Zeil

Last modified: Feb 27, 2014

1. Be Smart: Generate and Check

Testing addContact

In our earlier test for addContact, we weren’t particularly thorough:

TEST_F (MailingListTests, addContact) {
  mlist.addContact (jones);
  EXPECT_TRUE (mlist.contains("Jones"));
  EXPECT_EQ ("Jones", mlist.getContact("Jones").getName());
  EXPECT_EQ (4, ml.size());
}

Missing functional case: adding a contact that already exists
Missing boundary/special case: adding to an empty container
Missing special cases: Adding to beginning or end of an ordered sequence

Adding Variety

Some of our concerns could be addressed by adding tests but varying the parameters:

TEST_F (MailingListTests, addExistingContact) {
  mlist.addContact (baker);
  EXPECT_TRUE (mlist.contains("Baker"));
  EXPECT_EQ ("Baker", mlist.getContact("Baker").getName());
  EXPECT_EQ (3, ml.size());
}

Generate and Check

A useful design pattern for producing well-varied tests is

for v: varieties of ADT {
   ADT x = generate(v);
   result = applyFunctionBeingTested(x);
   check (x, result); 
}

generate and check could be separate functions or in-line
- depends on whether you can re-use in multiple tests
Most common “variety” would be size
- Could also be different constructors (in which case you might not literally have a loop:

   ADT x (constructor1);
   result = applyFunctionBeingTested(x);
   check (x, result); 

   ADT x2 (constructor2,  ... );
   result = applyFunctionBeingTested(x2);
   check (x2, result);

Example: addContact

A more elaborate fixture will aid in generating mailing lists of different sizes:

fixture.cpp

// The fixture for testing class MailingList.
class MailingListTests  {
public:
  Contact jones;
  vector<Contact> contacts;
  
  MailingListTests() {
    jones = Contact("Jones", "21 Penn. Ave.");
    contacts.clear();
    contacts.push_back (Contact ("Baker", "Drury Ln."));
    contacts.push_back (Contact ("Holmes", "221B Baker St."));
    contacts.push_back(jones);
    contacts.push_back (Contact ("Wolfe", "454 W. 35th St."));
  }

  ~MailingListTests() {
  }


  MailingList generate(int n) const
  {
    MailingList m;
    for (int i = 0; i < n; ++i)
      m.addContact(contacts[i]);
    return m;
  }

};

Example: addContact - many sizes

testAdd1.cpp

BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) {
  for (unsigned sz = 0; sz < contacts.size(); ++sz)
    {
      MailingList ml0 = generate(sz);
      MailingList ml (ml0);
      bool alreadyContained = ml.contains("Jones");
      ml.addContact (jones);
      BOOST_CHECK (ml.contains("Jones"));
      BOOST_CHECK_EQUAL ("Jones", ml.getContact("Jones").getName());
      if (alreadyContained)
      BOOST_CHECK_EQUAL (ml.size(), sz);
      else
    BOOST_CHECK_EQUAL (ml.size(), sz+1);
    }
}

Here we generate mailing lists of a variety of sizes and add Jones to them
At larger sizes, Jones is already in the list – functional case covered
sz == 0 covers one of our boundary/special cases

Example: addContact - ordering

testAdd2.cpp

BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) {
  for (unsigned sel = 0; sel < contacts.size(); ++sel)
    {
      Contact& toAdd = contacts[sel];
      const Name& nameToAdd = contacts[sel];
      for (unsigned sz = 0; sz < contacts.size(); ++sz)
        {
          MailingList ml0 = generate(sz);
          MailingList ml (ml0);
          bool alreadyContained = ml.contains(nameToAdd);
          ml.addContact (toAdd);
          BOOST_CHECK (ml.contains(nameToAdd));
          BOOST_CHECK_EQUAL (nameToAdd, ml.getContact(nameToAdd).getName());
          if (alreadyContained)
            BOOST_CHECK_EQUAL (ml.size(), sz);
          else
            BOOST_CHECK_EQUAL (ml.size(), sz+1);
        }
    }
}

We an also explore adding to different positions (beginning, middle, end)
- by varying which element we add

2. Be Thorough: Mutators and Accessors

Mutators and Accessors

To test an ADT, divide the public interface into

mutators: functions that alter the value of the object
accessors: functions that “look at” the current value of an object
- Occasional functions will fall in both classes

Organizing ADT Tests

The basic procedure for writing an ADT unit test is to

Consider each mutator in turn.
Write a test that begins by applying that mutator function.
Then consider how that mutator will have affected the results of each accessor.
Write assertions to test those effects.

Commonly, each mutator will be tested in a separate function.

2.1 Example: Unit Testing of the MailingList Class

Example: Unit Testing of the MailingList Class

mailinglist.h

#ifndef MAILINGLIST_H
#define MAILINGLIST_H

#include <iostream>
#include <string>

#include "contact.h"

/**
   A collection of names and addresses
*/
class MailingList
{
public:
  MailingList();
  MailingList(const MailingList&);
  ~MailingList();

  const MailingList& operator= (const MailingList&);

  // Add a new contact to the list
  void addContact (const Contact& contact);

  // Does the list contain this person?
  bool contains (const Name&) const;

  // Find the contact
  const Contact& getContact (const Name& nm) const;
  //pre: contains(nm)

  // Remove one matching contact
  void removeContact (const Contact&);
  void removeContact (const Name&);

  // combine two mailing lists
  void merge (const MailingList& otherList);

  // How many contacts in list?
  int size() const;


  bool operator== (const MailingList& right) const;
  bool operator< (const MailingList& right) const;

private:

  struct ML_Node {
    Contact contact;
    ML_Node* next;

    ML_Node (const Contact& c, ML_Node* nxt)
      : contact(c), next(nxt)
    {}
  };

  ML_Node* first;
  ML_Node* last;
  int theSize;

  // helper functions
  void clear();
  void remove (ML_Node* previous, ML_Node* current);

  friend std::ostream& operator<< (std::ostream& out, const MailingList& addr);
};

// print list, sorted by Contact
std::ostream& operator<< (std::ostream& out, const MailingList& list);


#endif

Look at our MailingList class.

Question: What are the mutators?

What are the accessors?

Answer:

Unit Test Skeleton

Here is the basic skeleton for our test suite.

skeleton.cpp

#define BOOST_TEST_MODULE MailingList test

#include "mailinglist.h"

#include <string>
#include <sstream>
#include <vector>

#include <boost/test/included/unit_test.hpp>

using namespace std;

Now we start going through the mutators.

Testing the Constructors

The first mutators we listed were the two constructors. Let’s start with the simpler of these.

Apply The Mutator

BOOST_AUTO_TEST_CASE ( constructor ) {
    MailingList ml;
      ⋮

First we start by applying the mutator.

Apply the Accessors to the Mutated Object

Then we go down the list of accessors and ask what we expect each one to return.

BOOST_AUTO_TEST_CASE ( constructor ) {
    MailingList ml;
    BOOST_CHECK_EQUAL (0, ml.size());
    BOOST_CHECK (!ml.contains("Jones"));
    BOOST_CHECK_EQUAL (ml, MailingList());
    BOOST_CHECK (!(ml < MailingList()));
}

It’s pretty clear, for example, that the size() of the list will be 0
contains() would always return false
The EQUAL test checks the operator==
We check for consistency of operator<

We can’t check the accessors

getContact
- can’t satisfy the pre-condition, and
operator<<
- behavior unspecified

Testing the Copy Constructor

testCons1.cpp

BOOST_FIXTURE_TEST_CASE ( copyConstructor,  MailingListTests) {
  for (unsigned sz = 0; sz < contacts.size(); ++sz)
    {
      MailingList ml0 = generate(sz);             ➊
      MailingList ml1 = ml0; // copy constructor  ➋
      shouldBeEqual(ml0, ml1);                    ➌
    }
  {
    // Minimal check for deep copy - changing one should not affect the other
    MailingList ml0 = generate(2);
    MailingList ml1 = ml0; // copy constructor
    ml1.addContact(jones);                        ➍
    BOOST_CHECK_EQUAL (2, ml0.size());
    BOOST_CHECK_NE (ml0, ml1);
  }
}

➊ We use the generate-and-check pattern.
➋ Here we invoke the function under test.
➌ The check function - we’ll look at this in a moment
➍ The second half of this is more subtle. I expect that copies are distinct. Once a copy is made, updating one object should not change the other.

A failure suggests that we have done an improper shallow copy.

The Check Function for Copying

shouldBeEqual.cpp

// The fixture for testing class MailingList.
class MailingListTests  {
 public:
  Contact jones;
  vector<Contact> contacts;
  
  MailingListTests() {
    ⋮
  
  void shouldBeEqual (const MailingList& ml0, const MailingList& ml1) const
  {
    BOOST_CHECK_EQUAL (ml1.size(), ml0.size());  ➊
    for (int i = 0; i < ml0.size(); ++i)         ➋
      {
    BOOST_CHECK_EQUAL(ml1.contains(contacts[i].getName()), ml0.contains(contacts[i].getName()));
    if (ml1.contains(contacts[i].getName()))
      BOOST_CHECK_EQUAL(ml1.getContact(contacts[i].getName()), ml0.getContact(contacts[i].getName()));
      }
    
    BOOST_CHECK_EQUAL (ml0, ml1);    ➌
    BOOST_CHECK (!(ml0 < ml1));
    BOOST_CHECK (!(ml1 < ml0));
    
    ostringstream out0;              ➌
    out0 << ml0;
    ostringstream out1;
    out1 << ml1;
    
    BOOST_CHECK_EQUAL (out0.str(), out1.str());
    
  }
  
};

➊ Sizes should be equal
➋ Boths lists should agree as to what they contain.
➌ Relational ops - should be equal and not less/greater
➍ Whatever they print, it should match

Testing the Assignment Operator

testAsst.cpp

BOOST_FIXTURE_TEST_CASE ( assignment,  MailingListTests) {
  for (unsigned sz = 0; sz < contacts.size(); ++sz)
    {
      MailingList ml0 = generate(sz);
      MailingList ml1;
      MailingList ml2 = (ml1 = ml0); // assignment   /*co1*/
      shouldBeEqual(ml0, ml1);                       /*co2*/
      shouldBeEqual(ml0, ml2);  // assignment returns a value
    }
  {
    // Minimal check for deep copy - changing one should not affect the other
    MailingList ml0 = generate(2);
    MailingList ml1;
    ml1 = ml0; // copy constructor
    ml1.addContact(jones);
    BOOST_CHECK_EQUAL (2, ml0.size());
    BOOST_CHECK_NE (ml0, ml1);
  }
}

Very similar to testing the copy constructor
➊ But remember that assignment operators no only change the value on the left, but also return a copy of the assigned value.
- We need to check both.
➋ Fortunately, we can re-use the check function developed for the copy constructor.

Testing addContact

testAdd.cpp

BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) {
  for (unsigned sel = 0; sel < contacts.size(); ++sel)
    {
      Contact& toAdd = contacts[sel];
      const Name& nameToAdd = toAdd.getName();
      for (unsigned sz = 0; sz < contacts.size(); ++sz)
        {
          MailingList ml0 = generate(sz);
          MailingList ml (ml0);
          bool alreadyContained = ml.contains(nameToAdd);
          ml.addContact (toAdd);
          BOOST_CHECK (ml.contains(nameToAdd));
          BOOST_CHECK_EQUAL (toAdd, ml.getContact(nameToAdd));
          if (alreadyContained)
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz);
              BOOST_CHECK_EQUAL (ml0, ml);
              BOOST_CHECK (!(ml0 < ml));
              BOOST_CHECK (!(ml < ml0));
            }
          else
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz+1);
              BOOST_CHECK_NE (ml0, ml);
              BOOST_CHECK ((ml0 < ml) || (ml < ml0));    // one must be true
              BOOST_CHECK (!((ml0 < ml) && (ml < ml0))); // ...but not both
            }
          ostringstream out;
          out << ml;
          BOOST_CHECK_NE (out.str().find(nameToAdd), string::npos);
        }
    }
}

Similar to version developed earlier, but now we go systematically through all the accessors
- Needed to add checks on relational operators and output operator

And so on…

We continue in this manner until done.

fullTest.cpp

#define BOOST_TEST_MODULE MailingList test

#include "mailinglist.h"

#include <string>
#include <sstream>
#include <vector>

#include <boost/test/included/unit_test.hpp>
//#define  BOOST_TEST_DYN_LINK 1
//#include <boost/test/unit_test.hpp>

using namespace std;

// The fixture for testing class MailingList.
class MailingListTests  {
 public:
  Contact jones;
  vector<Contact> contacts;
  
  MailingListTests() {
    jones = Contact("Jones", "21 Penn. Ave.");
    contacts.clear();
    contacts.push_back (Contact ("Muffin Man", "Drury Ln."));
    contacts.push_back (Contact ("Holmes", "221B Baker St."));
    contacts.push_back(jones);
    contacts.push_back (Contact ("Wolfe", "454 W. 35th St."));
  }
  
  ~MailingListTests() {
  }
  
  
  MailingList generate(int n) const
  {
    MailingList m;
    for (int i = 0; i < n; ++i)
      m.addContact(contacts[i]);
    return m;
  }
  
  void shouldBeEqual (const MailingList& ml0, const MailingList& ml1) const
  {
    BOOST_CHECK_EQUAL (ml1.size(), ml0.size());
    for (int i = 0; i < ml0.size(); ++i)
      {
        BOOST_CHECK_EQUAL(ml1.contains(contacts[i].getName()), ml0.contains(contacts[i].getName()));
        if (ml1.contains(contacts[i].getName()))
          BOOST_CHECK_EQUAL(ml1.getContact(contacts[i].getName()), ml0.getContact(contacts[i].getName()));
      }
    
    BOOST_CHECK_EQUAL (ml0, ml1);
    BOOST_CHECK (!(ml0 < ml1));
    BOOST_CHECK (!(ml1 < ml0));
    
    ostringstream out0;
    out0 << ml0;
    ostringstream out1;
    out1 << ml1;
    
    BOOST_CHECK_EQUAL (out0.str(), out1.str());
    
  }
  
};

BOOST_AUTO_TEST_CASE ( constructor ) {
  MailingList ml;
  BOOST_CHECK_EQUAL (0, ml.size());
  BOOST_CHECK (!ml.contains("Jones"));
  BOOST_CHECK_EQUAL (ml, MailingList());
  BOOST_CHECK (!(ml < MailingList()));
}


BOOST_FIXTURE_TEST_CASE ( copyConstructor,  MailingListTests) {
  for (unsigned sz = 0; sz < contacts.size(); ++sz)
    {
      MailingList ml0 = generate(sz);
      MailingList ml1 = ml0; // copy constructor
      shouldBeEqual(ml0, ml1);
    }
  {
    // Minimal check for deep copy - changing one should not affect the other
    MailingList ml0 = generate(2);
    MailingList ml1 = ml0; // copy constructor
    ml1.addContact(jones);
    BOOST_CHECK_EQUAL (2, ml0.size());
    BOOST_CHECK_NE (ml0, ml1);
  }
}

BOOST_FIXTURE_TEST_CASE ( assignment,  MailingListTests) {
  for (unsigned sz = 0; sz < contacts.size(); ++sz)
    {
      MailingList ml0 = generate(sz);
      MailingList ml1;
      MailingList ml2 = (ml1 = ml0); // assignment
      shouldBeEqual(ml0, ml1);
      shouldBeEqual(ml0, ml2);  // assignment returns a value
    }
  {
    // Minimal check for deep copy - changing one should not affect the other
    MailingList ml0 = generate(2);
    MailingList ml1;
    ml1 = ml0; // copy constructor
    ml1.addContact(jones);
    BOOST_CHECK_EQUAL (2, ml0.size());
    BOOST_CHECK_NE (ml0, ml1);
  }
}


BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) {
  for (unsigned sel = 0; sel < contacts.size(); ++sel)
    {
      Contact& toAdd = contacts[sel];
      const Name& nameToAdd = toAdd.getName();
      for (unsigned sz = 0; sz < contacts.size(); ++sz)
        {
          MailingList ml0 = generate(sz);
          MailingList ml (ml0);
          bool alreadyContained = ml.contains(nameToAdd);
          ml.addContact (toAdd);
          BOOST_CHECK (ml.contains(nameToAdd));
          BOOST_CHECK_EQUAL (toAdd, ml.getContact(nameToAdd));
          if (alreadyContained)
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz);
              BOOST_CHECK_EQUAL (ml0, ml);
              BOOST_CHECK (!(ml0 < ml));
              BOOST_CHECK (!(ml < ml0));
            }
          else
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz+1);
              BOOST_CHECK_NE (ml0, ml);
              BOOST_CHECK ((ml0 < ml) || (ml < ml0));    // one must be true
              BOOST_CHECK (!((ml0 < ml) && (ml < ml0))); // ...but not both
            }
          ostringstream out;
          out << ml;
          BOOST_CHECK_NE (out.str().find(nameToAdd), string::npos);
        }
    }
}

BOOST_FIXTURE_TEST_CASE (removeContact, MailingListTests) {
  for (unsigned sel = 0; sel < contacts.size(); ++sel)
    {
      Contact& toAdd = contacts[sel];
      const Name& nameToAdd = toAdd.getName();
      for (unsigned sz = 0; sz < contacts.size(); ++sz)
        {
          MailingList ml0 = generate(sz);
          MailingList ml (ml0);
          bool alreadyContained = ml.contains(nameToAdd);
          ml.removeContact (toAdd);
          BOOST_CHECK (!ml.contains(nameToAdd));
          if (!alreadyContained)
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz);
              BOOST_CHECK_EQUAL (ml0, ml);
              BOOST_CHECK (!(ml0 < ml));
              BOOST_CHECK (!(ml < ml0));
            }
          else
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz-1);
              BOOST_CHECK_NE (ml0, ml);
              BOOST_CHECK ((ml0 < ml) || (ml < ml0));    // one must be true
              BOOST_CHECK (!((ml0 < ml) && (ml < ml0))); // ...but not both
            }
          ostringstream out;
          out << ml;
          string outs = out.str();
          BOOST_CHECK_EQUAL (outs.find(nameToAdd), string::npos);
        }
    }
}

BOOST_FIXTURE_TEST_CASE (removeContactByName, MailingListTests) {
  for (unsigned sel = 0; sel < contacts.size(); ++sel)
    {
      Contact& toAdd = contacts[sel];
      const Name& nameToAdd = toAdd.getName();
      for (unsigned sz = 0; sz < contacts.size(); ++sz)
        {
          MailingList ml0 = generate(sz);
          MailingList ml (ml0);
          bool alreadyContained = ml.contains(nameToAdd);
          ml.removeContact (nameToAdd);
          BOOST_CHECK (!ml.contains(nameToAdd));
          if (!alreadyContained)
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz);
              BOOST_CHECK_EQUAL (ml0, ml);
              BOOST_CHECK (!(ml0 < ml));
              BOOST_CHECK (!(ml < ml0));
            }
          else
            {
              BOOST_CHECK_EQUAL (ml.size(), (int)sz-1);
              BOOST_CHECK_NE (ml0, ml);
              BOOST_CHECK ((ml0 < ml) || (ml < ml0));    // one must be true
              BOOST_CHECK (!((ml0 < ml) && (ml < ml0))); // ...but not both
            }
          ostringstream out;
          out << ml;
          BOOST_CHECK_EQUAL (out.str().find(nameToAdd), string::npos);
        }
    }
}

BOOST_FIXTURE_TEST_CASE ( merging,  MailingListTests) {
  
  MailingList ml0;
  ml0.addContact(contacts[0]);
  ml0.addContact(contacts[1]);
  ml0.addContact(contacts[2]);
  
  MailingList ml1;
  ml1.addContact(contacts[2]);
  ml1.addContact(contacts[3]);
  
  ml1.merge(ml0);
  shouldBeEqual(ml1, generate(4));
}

Yes, BTW, I did discover quite a few bugs in my MailingList code while putting this together.

2.2 Testing for Pointer/Memory Faults

Most destructors simply delete pointers, and pointer issues are particularly difficult to test and debug.

One way that pointer/memory use can be tested is to employ specialized tools that replace the system functions for allocating and deallocating memory (alloc and free) with special versions that
- keep track of all blocks of memory that have been allocated,
- whether those blocks have been freed,
- whether any block has been freed more than once, etc.

Tools for Testing Pointer/Memory Faults

Purify is a well known commercial package for this purpose, but is not cheap.
I have had good results with a free tool called LeakTracer.

3. Be Independent:

A true unit test should not depend on other modules of the program
- The *unit frameworks we have looked at provide drivers.
- But scaffolding can also involve stubs.
  - Which are often harder to write
We could
- Wait until integration test
- Prepare stubs for unit testing and accept that they can only be rough approximations.
- Create stubs for unit testing and replace them later when the lower-level modules are ready.

Is there a virtue to independence?

Integration tests are not independent.
- Can lead to slower testing.
- Can complicate debugging
- But are certainly more realistic that unit tests with simple stubs
Increasingly, *unit frameworks are providing support for stubbing.
- We’ll cover this in the next lesson on mock objects.

4. Be Pro-active: Write the Tests First

Ideally, we have made it easier to write self-checking unit tests than to write the actual code to be tested.

The goal is to encourage a philosophy of writing the tests before wirting the code.

Debugging: How Can You Fix What You Can’t See?

The test-first philosophy is easiest to understand in a maintenance/debugging context.

Before attempting to debug, write a test that reproduces the failure.
How else will you know when you’ve fixed it?
From a practical point of view, debugging generally involves running the buggy input, over and over, while you add debugging output or step through with a debugger.
- You want that to be as easy as possible.
- Doing this at the unit level, instead of in the context of the entire system, is often a dramatically better use of your time and effort.

Test-Writing as a Design Activity

Every few years, software designers rediscover the principle of writing tests before implementing code.

Agile and TDD (Test-Driven Development) are just the latest in this long chain.

Writing tests while “fresh” yields better tests than when they are done as an afterthought
Thinking about boundary and special values tests helps clarify the software design
- Reduces the number of eventual faults actually introduced
Encourages designing for testability
- Making sure that the interface provides the “hooks” you need to do testing in the first place.