 |
Table of Contents
Testing traditionally can be conducted in three styles
Black-Box testing
Try to choose "smart" tests based on the requirements, without looking at the code.
White-Box testing
Try to choose "smart" tests based on the structure of the code, with minimal reference to the requirements.
Random testing
Try to use directed random selection to choose tests that are "representative" of how the program will be used.
Black-box and white-box testing are complementary
projects should combine elements of each
Black-box tests
can be designed earlier
are better at catching errors of omission and errors in design
White-box tests
must await the development of the code
are better at catching errors of commission and errors after design
The most basic of all white-box methods, statement coverage is the selection of tests so that every statement has been executed at least once.
does not have to happen in a single test - as long as one test in the suite executes a statement, it's covered
How would you test this function so that every statement is covered?
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
int loc;
for (loc = 0; loc < listLength; loc++)
if (list[loc] == searchItem)
return loc;
return -1;
}
Can we cover this statement with a test?
int BidderCollection::add (const Bidder& value)
// Adds this bidder
//Pre: getSize() < getMaxSize()
{
if (size < MaxSize)
{
addToEnd (elements, size, value);
return size - 1;
}
else
{
cerr << "BidderCollection::add - collection is full" << endl;
exit(1);
}
}
During unit test, certainly.
During an integration or system test, probably not.
Typical of code inserted as "defensive programming"
We can check whether we have covered statements by adding debugging output.
add a unique output to each block of "straight line" code
#define COVER cerr << "Block " << __FILE__ << ":" << __LINE__ << endl template <typename T> int seqSearch(const T list[], int listLength, T searchItem) { COVER; int loc; for (loc = 0; loc < listLength; loc++) if (list[loc] == searchItem) { COVER; return loc; } COVER; return -1; }
However, g++ and other compilers offer tools that can automate this process.
We can go a long way by simply being aware of the idea of statement coverage when we develop our tests.
Recognize that we want tests to cover every branch
But as programs get more complicated, even experienced testers do a poor job of stmt coverage
Automatic coverage tools can help
Coverage tools need to understand the control flow of your code
typically have to duplicate most of the work of a compiler
often cost big $$
It's a lot easier to do this if the compiler provides support in the first place
The gcc/g++ suite includes a coverage tool,
gcov
We will look at doing a unit test of the three search functions in arrayUtils.h
First, we need a test driver. We can go with either of two styles.
In one, we write a main program that reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.
Alternatively this version
uses no external input but instead contains code to
construct the data we need, then asserts that
the search functions produce the expected results.
For this example we'll use the first style.
To use gcov, we compile with special options
-fprofile-arcs -ftest-coverage
In Unix, we would add these to the compilation commands or add them to the C++ options in a makefile such as this one.
In Code::Blocks, we can add these to "Project ->
Build Options -> Compiler Settings -> Other
options", and also by going to ""Project -> Build
Options -> Linker Settings" and, under "Link
libraries", adding the libgcov.a file found
inside the Code::Blocks installation directory, probably
under MingW\lib\gcc\mingw32\3.4.5 (the
version number at the end may vary).
When the code has been compiled, in addition to the usual
files there will be several files with endings like
.gcno These hold data on where the statements and
branches in our code are.
Run your tests normally.
As you test, a *.gcda file will accumulate
data on which statements have been covered.
Run gcov mainProgram
the immediate output will be a report on the percentages of statements covered in each source code file.
Also creates a detailed report for each source code file. e.g.,
-: 69:template <typename T>
-: 70:int seqSearch(const T list[], int listLength, T searchItem)
-: 71:{
1: 72: int loc;
-: 73:
2: 74: for (loc = 0; loc < listLength; loc++)
2: 75: if (list[loc] == searchItem)
1: 76: return loc;
-: 77:
#####: 78: return -1;
-: 79:}
Lists number of times each statement has been executed
Lists #### if a statement has never been executed
Focus on these as you choose additional tests
Code::Blocks users will find gcov in the same
directory as the g++ executables. The easiest way
to handle this is to start a Windows cmd window, cd
to the directory where Code::Blocks has placed your compiled
code, then do
pathToExecutables\gcov mainProgram Branch coverage is a requirement that, for each branch in the program (e.g., if statements, loops), each branch have been executed at least once during testing. (It is sometimes also described as saying that each branch condition must have been true at least once and false at least once during testing.)
How is branch coverage different from statement coverage? If you have covered every statement in
if (x > 0)
y = x;
else
y = -x;or in
while (z > 0)
--z;
have you not covered all the branches as well?
Only difference occurs when branches are "empty":
y = x;
if (x < 0)
y = -x;or in
while (z-- > 0);
In these cases branch coverage would require tests in which
x and z are positive. Statement coverage would
not.
How would you test this function so that every branch is covered?
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
int loc;
for (loc = 0; loc < listLength; loc++)
if (list[loc] == searchItem)
return loc;
return -1;
}
Again, just being aware of the idea of branch coverage can help guide our tests.
Recognize that we want tests to cover every branch
But as programs get more complicated, even experienced testers do a poor job of stmt/branch coverage
Automatic coverage tools can help
gcov can report on branches taken. Just
add options to the gcov command:
gcov -b -c mainProgram gcov reports
Number of times each function call successfully returned
# of times a branch was executed (really how many times the branch condition was evaluated)
and # times each branch was taken
For branch coverage, this is the relevant figure
A "branch" is anything that causes the code to not continue on in straight-line fashion
Branch listed right after an "if" is the "branch" that jumps around the "then" part to go to the "else" part.
&& and ||
operators introduce their own branches
Other branches may be hidden
Contributed by calls to inline functions
Or just a branch generated by the compiler's choice of code generation
-: 84:template <typename T>
-: 85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
-: 86:{
1: 87: int loc = 0;
-: 88:
1: 89: while (loc < listLength && list[loc] < searchItem)
branch 0 taken 0
call 1 returns 1
branch 2 taken 0
branch 3 taken 1
-: 90: {
#####: 91: ++loc;
branch 0 never executed
-: 92: }
1: 93: if (loc < listLength && list[loc] == searchItem)
branch 0 taken 0
call 1 returns 1
branch 2 taken 0
1: 94: return loc;
branch 0 taken 1
-: 95: else
#####: 96: return -1;
-: 97:}
Report is organized by basic blocks, straight-line sequences of code terminated by a branch or a call
Hard to map to specific source code constructs
lowest-numbered branch is often the leftmost condition
Fact of life that compilers insert branches and calls that are often invisible to us
Various definitions of loop coverage exist.
One of the more useful suggests that a loop is covered if in at least one test the body was executed 0 times, and if in some test the body was executed exactly once, and if in some test the body was executed more than once.
A good idea to keep in mind, but harder to monitor.
|
Discuss This Page: