1.2 Choosing Test Data

Strategies for Choosing Tests

Testing traditionally can be conducted in three styles

• Black-Box testing

  • Try to choose “smart” tests based on the requirements, without looking at the code.

• White-Box testing

  • Try to choose “smart” tests based on the structure of the code, with minimal reference to the requirements.

• Random testing

  • Try to use directed random selection to choose tests that are “representative” of how the program will be used.

We’ll discuss the black-box testing of these today and white-box later this semester.

Random testing is not used as often. It sometimes is useful as a way check against unwarranted assumptions by the developers and testers (“Oh, I never thought of even trying that!”). But its primary use is in large projects where statistics collected during random testing are plugged into models that are used to predict how reliable the software will be if released in its current state or how much longer we will need to test before we reach a desired level of reliability.

2 Black Box Testing

Black-Box Techniques

There are a number of well-known BB strategies.

• Representative Inputs

• Functional Coverage

• Boundary-Values Testing

• Special-Values Testing

A good test BB suite will include all of these.

In fact, these should become second nature to most programmers.

2.1 Representative Inputs

Choose at least one test that is a “typical” input to the system when in real use.

• This may require some study of how people will use the system.
  • If you are replacing an existing system, or automating a task that people have been doing manually, you may be able to collect “real” inputs.

Example: the Auction Program

2.2 Test Specifications

Software development projects manage testing via a couple of standard documents:

• A test plan is a document describing what will be tested, who will do it, and the procedures they will follow.

• A test case specification or test specification describes the test cases for an item to be tested.

• These are often confused with one another.

  • Test plans are for management.

  • Test specs are for the software developers.

2.3 Functional Coverage

This is a bit of a misnomer. the “function” in “Functional Coverage” has nothing to do with the idea of a C++ function. Instead it really pertains more to “functionality”.

If you look though most program requirements, you’ll see that the program is probably supposed to carry out a variety of different behaviors under various circumstances. Sometimes these behaviors are described in terms of input conditions:

For any input $x$, if $x \geq 0$, return $x$ but if $x < 0$, return $-x$.

Here we have clearly drawn a distinction between different behaviors depending upon the input values.

In other cases, distinct behaviors are most easily identified by looking at the output requirements:

This program reads a file name from the input. It should print a single line of output

filename file-size-in-kilobytes

if the file exists but should print

filename does not exist.

if the file cannot be found.

Here we can tell from the distinct output specifications that we have two separate behaviors going on.

Now, from a purely theoretical point of view, distinct behaviors must have both distinct input pattern and distinct output patterns. I could just as easily argue that, in the above example, the two behaviors are associated with the input cases of “the input named an existing file” and “the input named a non-existent file”. But sometimes the people writing the requirements will find it easier to focus on the input conditions and sometimes on the output cases. As a tester (and coder), you have to roll with whatever they give you.

Sometimes, a behavior may be described in purely internal terms:

Apply the formula

x = x - (x*x - N) / 2

up to 1000 times until x changes by no more than $\pm 0.001$. If, after 1000 iterations, x is still changing by more than that, abort the program with message

Calculation does not converge.

In this case, if x is not an input but some value computer internally, it might be quite difficult to figure out what inputs would lead to this “does not converge” behavior. Nonetheless, as a tester, you will want to try to do so.

Functional Coverage

Choose at least one test that covers each distinct “behavior” described in the requirements.

• Different functions performed by the program

• Different types or ranges of input that get treated or described separately.

• Different types or ranges of output that get treated or described separately.

Don’t make the common mistake of that different parts of a program’s calculations or outputs are distinct behaviors. For example, given a requirement

This program reads a file name from the input.

It should print that file name.

If the file exists, it should then print the size of the file (in kilobytes), on the same line.

If the file does not exist, it should print “ does not exist.” on the same line as the file name.

We only have two distinct behaviors here, not three. The printing of the file name is not a distinct behavior from the behavior of printing the file size or the “does not exist” message. It’s simply a component of the behavior in each of the other two cases. How do we know? Because there is no distinct set of inputs on which file names are (and are not) printed.

Example: Auction Functional Coverage

Question:

2.4 Boundary Values Testing

A.k.a., Extremal Values Testing

Choose as test data the largest and the smallest values for each input and for each “functional behavior” range

Seeking Boundaries

• A common mistake is to simply look at the input data and to try to choose the largest and smallest inputs possible.

  • Nothing wrong with using that data

  • But it shortchanges the test set quite a bit.

• Instead, seek the boundaries of the functional test cases that you specified earlier.

Sample Boundary Tests

Suppose that we are testing an implementation of the absolute value function abs(x)

• Distinct behaviors for $x \geq 0$ and $x < 0$

  • So, during functional coverage, we might use tests $x=12$ and $x=-1015$

• But boundary value testing would suggest that we look at

  • An extremely large $x$ and $x=0$: the boundaries of the $x \geq 0$ behavior
  • An extremely large negative number and $x=-1$: the boundaries of the $x < 0$ behavior

The test cases $0$ and $-1$ are useful because they give us a chance of detecting common errors like using the wrong relational operator.

Notice that, if we have done a thorough job of listing the functional cases for a program, then we will get the “largest and smallest possible inputs” cases automatically when we list out the largest and smallest cases for each distinct behavior.

Example: Auction Extremal Values

We had functional cases

• Highest bid for some item qualifies to win.

• Highest bid for some item is too late but otherwise qualifies to win. Concentrating on the time input, we would focus on the boundary between being on time and too late:

• Bids can be

  • exactly on time
  • one second late

Example: Auction Extremal Values (cont.)

We also have some explicit limits on the legal inputs, e.g.:

• #items is non-negative
• auction end time: hours is 00..23, minutes is 00..59, seconds is 00..59

This leads to some additional boundary cases:

• #items of zero

• auction ending at midnight (00:00:00)

• auction ending one second before midnight (23:59:59)

Continuing the Auction Program Test Specification

2.5 Special Values Testing

Choose as test data those certain data values that just tend to cause trouble. Programmers eventually develop a sense for these. They include

• For integers: -1, 0, 1

• For floating point numbers: -e, 0, +e, where “e” is a very small number

• For strings: the empty string, strings containing only blanks, strings containing no alphabetic characters

What is Special?

• As we move to other data structures, we may develop a suspicion of other special values, e.g.,

  • For times of the day: midnight, noon

  • For containers of data: an empty container

Special values and Boundary values often overlap. That’s OK.

One of my favorite stories about bugs that (should have) been caught by boundary and special values testing comes from early testing of the software controllers for the F-16 jet fighter. During fight simulations (thankfully!), one pilot decided to fly the simulated plane across the equator. A sign error in handling southern latitudes caused the control software to decide that the plane was upside down, and to attempt to roll the plane over to compensate.

Example: Auction Special Cases

Special Values

1. Number of items is zero

• Number of items is one

• Item name has no alphabetic characters

• Item reserve price is zero
• Auction ends at midnight

• Auction ends at noon

• Number of bidders is zero

• Number of bidders is one

• Bidder's balance is zero

• Bidder’s name has no alphabetic characters

• Number of bids is zero

• Number of bids is one

• Amount bid is zero

• Time of bid is midnight

• Time of bid is noon

2.6 Completed Test Specification

After eliminating duplicates from the preceding list,

3 Illegal Inputs

Earlier, we observed that the input requirement:

#items is non-negative

led to a boundary value test case:

• #items of zero

I did not add a test case

• #items is $-1$ Why not?

If we have an input requirement

#items is non-negative

then the input #items == -1 is illegal.

• You cannot test a program on illegal inputs.

Notice that I did not say “should not”. I said “cannot”. It can’t be done.

Therefore …

• Trying to test a program on illegal inputs is just a waste of time and effort.

You may have been told something differently by other teachers or read something different elsewhere.

I don’t care.

They are either wrong, or you misunderstood what they were talking about.

There are people who make statements like “a good program never crashes” or “a good program should always behave sensibly on any input”. Setting aside the fact that the first is impossible and the second relies on a definition of “sensible” that no two people will ever agree upon, these kinds of statements are really aimed at how to write a good requirements statement for programs. They have nothing at all to do with how we code programs or how we test them.

3.1 Why CAN’T we test illegal inputs?

• A test is useless if you can’t tell whether the program passes or fail is.

• By definition, a program can do anything on an illegal input.

  • It could print an error message and abort

  • It could print an error message and try to keep running

  • It could pop up a menu asking the operator what it should do.

  • It could send email to all of your friends complaining about the lousy input that you tried to force it to process.

  • It could contact the nearest SAC base and call in an airstrike on your keyboard.

On an illegal test input, no matter the program actually does, it passes the test by definition.

3.2 Illegal versus Unusual Inputs

Don’t confuse illegal inputs with unusual inputs

• If I ask you to write a function mySqrt(double x) that

  computes the square root of any non-negative x

  then negative inputs are illegal.

  • The function has no specified behavior in that case. Any program that calls this function must make sure it never supplies negative input values.

• If I ask you to write a function mySqrt(double x) that

  computes the square root of any non-negative x but returns -1.0 when given a negative x

  then negative inputs are not illegal.

  • We might expect them to be rare, but the behavior of the function is fully specified in that case,
  • so it’s perfectly OK for programs to call the function with negative numbers.

3.3 How do you test a program on …

• … an illegal input?

  • You can’t. Whatever the program does is correct behavior for that program.

• … an unusual input?

  • Check to be sure that it does whatever the requirements document says.
  • Such tests should emerge naturally from functional coverage
    • Each “kind” of unusual input is a distinct behavior.

4 Some General Guidelines for Testing

4.1 Choosing the Test Data

Within the limits of the test case specification,

• Choose the simplest inputs you can

  • Easy to come up with the input

  • Easy to show it satisfies the test case

  • Easy to figure out what the correct output is supposed to be

  • Easy to see if the output actually is correct

• OK to overlap test cases on the same input set

  • as long as they don’t interfere with or “hide” one another

  • but usually better to have lots of simple tests than a few large ones

• KISS!

  • Because we all get a little bit “stupid” after staring at the screen for hours

4.2 Conducting Tests

• Get out of the habit of typing all your tests by hand

  • Discourages people from running more than a few tests

  • Makes it hard to repeat tests

• Alternatives:

  • redirection (CS252, but also possible in Windows)

  • copy-and-paste from text file into running program

  • redesign program to allow file names in command line

Many of these are covered in the earlier labs.