CS 250 Computer Programming and Problem Solving - FALL 1998 

[ Home | Syllabus | Notes | Glossary | CS250 WEB Version Home Page ]

Testing Software

Software Validation

Validation is the set of activities conducted to convince ourselves that the software performs as intended.

On successful large projects, validation may consume 50\% to 75\% of the total effort from Requirements Analysis till release.

Common Forms of Validation

• Desk Checking
• Code Reviews
• Testing
• Reliability Measurement
• Debugging

Desk Checking

An exercise conducted by the individual programmer.

• ``Playing computer'' with the aid of a listing. Values of variables are tracked using pencil and paper as the programmer moves step-by-step through the code.
• Can be done with pseudocode or ``walkthroughs'' of other logic tools.
• A good way to find fundamental flaws in algorithms, especially before actually writing the code.
• Useful in checking the results of an intended change during debugging.

Code Reviews

• Conducted by a team, often including other programmers who will use the code.
• Led by someone other than the programmer.
• The programmer is asked to explain the code to the review team. The team may ask questions, and may decide to try a ``walkthrough'' of critical sections of the code.
• A team member (other than the programmer and the leader) serves as ``secretary'' to record any problems noted.
• Emphasis is on discovery of problems. Possible fixes are not discussed.

Reliability Models

A statistical process for estimating the reliability of software.

For example, suppose that we track the amount of time we can run
without failures over a series of bug fixes:

1. Bug 1 detected after 1 min.
2. After fixing bug 1, we ran for 10 min before finding bug 2.
3. After fixing bug 2 we ran for 2 hrs before finding bug 3.

We might guess that this program will run, on average, for over 20 hrs. before failing due to some unknown bug 4.

Reliability Models (cont)

Another approach is based upon deliberately "seeding" faults into the code. The code is tested, and the faults that are found are examined to see how many were "seeded" and how many were "real".

Suppose we seed a program with 10 faults. Someone else tests the code and finds 20 faults, 5 of which were ones that we seeded.

Since only 25% of the seeded faults were found, we estimate that the test data is so poor that
it has probably found only 25% of the real faults.

We conclude that 45 "real" faults remain in the code.

Testing

Testing is the act of executing a program with selected data to uncover bugs.

Fault

A defect in the source code.

Failure

An incorrect behavior or result.

Error

A mistake by the programmer, designer, etc., that led to the the fault.

Industry figures of 1-3 faults per 100 statements are quite common.

Testing by Time and by Scale

Unit Test

Tests of individual subroutines and modules, usually conducted by the programmer.

Integration Test

Tests of ``subtrees'' of the total project hierarchy chart --- groups of subroutines calling each other. Integration Testing is generally a team responsibility.

System Test

Test of the entire system, supervised by team leaders or by V&V specialists.

Many companies have independent teams for this purpose.

Regression Test

Unit/Integration/System tests that are repeated after a change has been made to the code.

Acceptance Test

A test conducted by the customers or their representatives to decide whether to purchase/accept a developed system.

Integration Testing

• How to test a module when you have some, but not all, of the rest of the program?
• Need to add scaffolding --- special code written solely for the purpose of testing.
• Integration testing can be performed top-down or bottom-up.
  

Top-Down Integration Testing

• Highest level routines (in hierarchy chart) added and tested first.
• Scaffolding consists of stubs --- ``dummy'' bodies for the lower level routines called by the ones under test.
  Stubs may:
  • be an (almost) empty body
  • contain a simple tracing cout
  • assign constants to output parameters and function return values
  • obtain output/return values by other means (e.g., read from keyboard/file)

Skeleton Programs

• Programs that are all scaffolding.
• A fill-in-the-blanks description of the high-level design.
• Each module/subroutine is represented by a stub having the proper formal parameter list.
• A skeleton program can be compiled and run, though the results not very interesting.
• Sometimes the main routines may contain minimal control flow to call each of its children in the hierarchy chart in an approximately normal order.

Testing Process

Oracle

The oracle is the procedure for determining whether a failure has occured on some test.

Possible oracles include

• human inspection of output
• automatic comparison against expected outputs (e.g., the regression log)
• automatic comparison against output of another program that does the same job.
• Special purpose specification-checking programs

"Partial" oracles consisting of debugging code within the program under test.

Test Plans

• Developed for each form of test (unit, integration, etc.)
• Must indicate
  • Scope of plan (which modules/routines does plan apply to, which form of testing, etc.)
  • Integration plan (as applicable)
  • How is test input data selected?
  • How is expected output determined?
  • Purpose of each test?
• Test plans may explicitly list each (input,~output) pair, or may simply indicate the test method/oracle form to be used.
• Test plans are developed in conjunction with other phases of the life cycle:

Testing Methods

• Test with ``real'' data
  Collected from past or by operating new system in parallel with old.
• Test with simulated data
  Random test generation attempting to match the ``distribution'' of the real data.
• Specification-based (a.k.a. Black-Box) testing
  Functional testing
  • Boundary-value testing
  • Special-Values testing
• Implementation-based (a.k.a. White-Box) testing
  Structural Testing
  • Statement Coverage
  • Branch Coverage
• Other techniques such as ``Mutation Testing'' (based on error seeding) are still unproven.

Functional Testing

Attempt to choose test data illustrating each distinct behavior or each
distinct class of inputs at least once.

Example: parentheses matching:

Matching ( )
Matching { }
Matching [ ]
Multiple nested groups
Multiple unnested groups
No parenthetical groups
Missing left paren
Missing right paren

Boundary-Values Testing

Choose data at the boundaries of a functional testing class or of the overall input domain.

Example: parentheses matching:

No parentheses
Empty input string
Very long input string
Matching right paren as last character
Left paren as first character

Special-Values Testing

Choose data reflecting ``special'' or troublesome cases.

Examples include choosing for each numeric input negative, zero, and positive values, testing each string input with empty and entirely blank strings, etc.

Statement Coverage

Require that every statement in the code be executed at least once during testing.

Special programs (``software tools'') will monitor this requirement for you.

Methods and Times

Copyright chris wild 1998.
For problems or questions regarding this website contact [Chris Wild (e-mail:wild@cs.odu.edu].
Last updated: November 04, 1998.