3.1 Black-Box Testing
Black-box (a.k.a. specification-based) testing chooses
tests without consulting the implementation.
- based simply upon our understanding of what the unit is supposed to
do.
One of the goals of black-box testing is to be sure
that every distinct behavior expected of a unit has been triggered in
at least one test. Another is to try to choose tests that are likely
to cause trouble, no matter what the actual algorithm is.
Some of the best-known techniques for choosing black-box tests focus
on the input values that will be supplied to the unit during testing.
Functional Coverage
a.k.a Equivalence partitioning
Boundary Values Testing
a.k.a., Extremal Values Testing
- Choose as test data the largest and the smallest values for each
input and for each “functional” case
- Often misunderstood as simply choosing largest & smallest possible inputs
- Assumes that we have started by attempting functional coverage
Special Values Testing
Choose as test data those certain data values that just tend to
cause trouble.
Programmers eventually develop a sense for these. They include
For integers: –1, 0, 1
For floating point numbers: -e, 0, +e, where “e” is a very
small number
For strings: the empty string, strings containing only
blanks, strings containing no alphabetic characters
What is “Special”?
3.2 White-Box Testing
White-Box (a.k.a. Implementation-based testing)
uses information from the implementation to choose tests.
Formally, most white-box testing involves coverage measures, E.g., statement coverage, branch coverage
For example, a common goal in white-box testing is to be sure that
every code statement is executed at least once by some
test. Another common goal is that, for every conditional branch (ifs,
loops, etc.) in the code, that on at least one test that condition
will have been true and on at least one test that condition will have
been false.
The difficulty with such white-box testing is that it’s almost
impossible to be sure you have met one of these goals unless you
have special testing software to track the execution of
statements and branches over the whole history of
testing. Lacking this software, it’s easier to do black-box
testing.
Informally, design tests specifically to exercise “tricky” parts of the code
However, there is a kind of informal white-box testing that we can
do. As programmers, we often write a bit of code and think to ourselves
“I wonder if this is really going to work?” That’s a good
time to pause and design a test that will check exactly that bit of
dicey code.
Common forms:
Structural Testing (a.k.a., “path testing” (not per your
text)
Designate a set of paths through the program that must be exercised
during testing.
- Statement Coverage
- Branch Coverage
- Cyclomatic coverage (“independent path testing”)
- Data-flow Coverage
Mutation testing
Perturbation testing
Statement Coverage
Require that every statement in the code be executed at least once
during testing.
Special programs (“software tools”) will monitor this requirement for you.
- gprof in GNU gcc/g++ suite
- Clover and JaCoCo for Java
Example
cin >> x >> y;
while (x > y)
{
if (x > 0)
cout << x;
x = f(x, y);
}
cout << x;
What kinds of tests are required for statement coverage?
Branch Coverage
Requires that every “branch” in the flowchart be tested at least once
- Equivalent to saying that each conditional stmt must be tested as both true and false
- Branch coverage implies Statement Coverage, but not vice versa
if X < 0 then
X := -X;
Y := sqrt(X);
Branch Coverage example
cin >> x >> y;
while (x > y)
{
if (x > 0)
cout << x;
x = f(x, y);
}
cout << x;
What kinds of tests are required for branch coverage?
Multi-Condition Coverage
a.k.a., Condition coverage
Various approaches to coping with boolean expressions,
particularly short-circuited ones.
Goal: given a boolean expression \( a \oplus b \) , where \( \oplus \)
could be & , && , |, etc., need at least one test where
- \( a \) is true and, had it been false, the value of \( a \oplus b \)
would change
- \( a \) is false and, had it been true, the value of \( a \oplus b \)
would change
- \( b \) is true and, had it been false, the value of \( a \oplus b \)
would change
- \( b \) is false and, had it been true, the value of \( a \oplus b \)
would change
For example, for the expression a & b, we would need the
combinations
| a |
b |
| true |
true |
| false |
true |
| true |
false |
Cyclomatic Coverage
(a.k.a “independent path coverage”, “path testing”)
A Control Flow Graph
- 1, 2, 3, 4, 12, 13
- 1, 2, 3, 5, 6, 11, 2, 12, 13
- 1, 2, 3, 5, 7, 8, 10, 11, 2, 12, 13
- 1, 2, 3, 5, 7, 9, 10, 11, 2, 12, 13
Cyclomatic Complexity
The number of independent paths in a program can be
discovered by computing the cyclomatic complexity (McCabe, 1976)
\[CC(G) = \mbox{Number}(\mbox{edges}) - \mbox{Number}(\mbox{nodes}) + 1\]
This is a popular metric for module complexity.
Actually pretty trivial: for structured programs with only binary
decision constructs, equals number of conditional statements +1
- relation to testing is dubious
- simply branch coverage hidden behind smoke and mirrors
Uniqueness
Sets of independent paths are not unique, nor is their size:
- 1,2,3,5,6,11,2,3,5,7,8, 10,11,2,3, 5,7,9,10,11, 2,12,13
- 1,2,3,4,12,13
Data-Flow Coverage
Attempts to test significant combinations of branches.
Any stmt i where a variable X may be assigned a new value is called a
definition of X at i: def(X,i)
Any stmt i where a variable X may be used/retrieved is called a
reference or use of
X at i: ref(X,i)
def-clear
- A path from stmt i to stmt j is
def-clear with respect to X if it contains no definitions of
X except possibly at the beginning (i) and end (j)
all-defs
The all-defs criterion requires that each definition
def(X,i) be tested some def-clear path to some reference
ref(X,j).
1: cin >> x >> y; d(x,1) d(y,1)
2: while (x > y) r(x,2), r(y,2)
3: {
4: if (x > 0) r(x,4)
5: cout << x; r(x,5)
6: x = f(x, y); r(x,6), r(y,6), d(x,6)
7: }
8: cout << x; r(x,8)
What kinds of tests are required for all-defs coverage?
all-uses
The all-uses criterion requires that each pair
(def(X,i), ref(X,j)) be tested using some def-clear path from
i to j.
1: cin >> x >> y; d(x,1) d(y,1)
2: while (x > y) r(x,2), r(y,2)
3: {
4: if (x > 0) r(x,4)
5: cout << x; r(x,5)
6: x = f(x, y); r(x,6), r(y,6), d(x,6)
7: }
8: cout << x; r(x,8)
What kinds of tests are required for all-uses coverage?
Mutation Testing
Given a program \(P\),
Mutation Testing (cont.)
Mutation Testing (cont.)
A set of test data is considered inadequate if it cannot
distinguish between the program as written (\(P\)) and programs that
differ from it by only a simple change.
- So if any mutants are still alive after running a set of tests, we
augment the tests until we can kill all the mutants.
Mutation Testing Problems
⋮ ⋮
X = Y; X = Y;
if (X > 0) then if (Y > 0) then}
⋮ ⋮
- Identifying these can be difficult (and cannot be automated)
Perturbation Testing
Perturbation testing (Zeil) treats each arithmetic expression
\( f(\bar{x}) \) in the code as if it had been modified by the addition of
an error term \( f(\bar{v}) + e(\bar{v}) \) , where \( v \) are the program
variables and \( e \) can be a polynomial
of arbitrarily high degree (can approximate almost any error)
