1.2. Statement Coverage

The most basic of all white-box methods, statement coverage is the selection of tests so that every statement has been executed at least once.

Example: statement coverage

How would you test this function so that every statement is covered?

template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
    int loc;

    for (loc = 0; loc < listLength; loc++)
        if (list[loc] == searchItem)
            return loc;

    return -1;
}

100% Coverage may not be Possible

Can we cover this statement with a test?

int BidderCollection::add (const Bidder& value)
//  Adds this bidder
//Pre: getSize() < getMaxSize()
{
  if (size < MaxSize)
    {
     addToEnd (elements, size, value);
     return size - 1;
    }
  else
    {
     <[+]>cerr << "BidderCollection::add - collection is full" << endl;<[-]>
     exit(1);
    }
}

  • During unit test, certainly.

  • During an integration or system test, probably not.

    • Typical of code inserted as "defensive programming"

Monitoring Statement Coverage

We can check whether we have covered statements by adding debugging output.

  • add a unique output to each block of "straight line" code

#define COVER cerr << "Block " << __FILE__ << ":" << __LINE__ << endl
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
    <[+]>COVER;<[-]>
    int loc;

    for (loc = 0; loc < listLength; loc++)
        if (list[loc] == searchItem)
            <[+]>{
             COVER;<[-]>
             return loc;
            <[+]>}<[-]>

    <[+]>COVER;<[-]>
    return -1;
}

However, g++ and other compilers offer tools that can automate this process.

Monitoring Statement Coverage with gcov

We can go a long way by simply being aware of the idea of statement coverage when we develop our tests.

  • Recognize that we want tests to cover every branch

  • But as programs get more complicated, even experienced testers do a poor job of stmt coverage

    • Automatic coverage tools can help

gcov
  • Coverage tools need to understand the control flow of your code

    • typically have to duplicate most of the work of a compiler

    • often cost big $$

  • It's a lot easier to do this if the compiler provides support in the first place

    • The gcc/g++ suite includes a coverage tool, gcov

Example: Unit Testing the Array Search Functions
  • We will look at doing a unit test of the three search functions in arrayUtils.h

  • First, we need a test driver. We can go with either of two styles.

    • In one, we write a main program that reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.

    • Alternatively this version uses no external input but instead contains code to construct the data we need, then asserts that the search functions produce the expected results.

  • For this example we'll use the first style.

Compiling for gcov

To use gcov, we compile with special options

  • -fprofile-arcs -ftest-coverage

    • In Unix, we would add these to the compilation commands or add them to the C++ options in a makefile.

    • In Code::Blocks, we can add these to "Project -> Build Options -> Compiler Settings -> Other options", and also by going to ""Project -> Build Options -> Linker Settings" and, under "Link libraries", adding the libgcov.a file found inside the Code::Blocks installation directory, probably under MingW\lib\gcc\mingw32\3.4.5 (the version number at the end may vary).

  • When the code has been compiled, in addition to the usual files there will be several files with endings like .bb, .bbg. These hold data on where the statements and branches in our code are.

Running Tests with gcov
  • Run your tests normally.

  • As you test, various *.da files will accumulate data on which statements have been covered.

Viewing Your Report
  • Run gcov mainProgram

    • the immediate output will be a report on the percentages of statements covered in each source code file.

    • Also creates a detailed report for each source code file. e.g.,

           -:   69:template <typename T>
           -:   70:int seqSearch(const T list[], int listLength, T searchItem)
           -:   71:{
           1:   72:    int loc;
           -:   73:
           2:   74:    for (loc = 0; loc < listLength; loc++)
           2:   75:        if (list[loc] == searchItem)
           1:   76:            return loc;
           -:   77:
       #####:   78:    return -1;
           -:   79:}
      

      • Lists number of times each statement has been executed

      • Lists #### if a statement has never been executed

        • Focus on these as you choose additional tests

  • Code::Blocks users will find gcov in the same directory as the g++ executables. The easiest way to handle this is to start a Windows cmd window, cd to the directory where Code::Blocks has placed your compiled code, then do

    pathToExecutables\gcov mainProgram 

Resetting the Report Data
  • Delete the .da files to reset all counts to zero.

    • e.g., if you change or remove tests from your test set