1 The Testing Process

The diagram here illustrates the steps involved in testing code.

• Beginning from an overall test plan (or test specification),

• we eventually seek to discover a collection of failures

  • These failures become the input to the process of debugging, where we seek to find the faults in the code responsible for those failures.

• Failure: An execution on which incorrect behavior occurs

• Fault: A defect in the code that (may) cause a failure

Related to these, though not part of our testing process:

• Error: A human mistake that results in a fault, or,

  • alternatively, the difference between the expected output and the actual output on a failed test.

1.1 The Steps in a Testing Process

A test plan (more properly, a test specification) describes a set of test cases.

• Test case: a general description of a required test

1. Derive inputs for each test case.

   In most cases, you will also need to record the expected outputs or behavior for your test inputs.

   The inputs and expected outputs may be recorded in a database of regression tests for later. But the most obvious use for the new inputs is to…

2. Execute the tests

   The test inputs are fed into the program being tested and the actual outputs collected.

3. Determine which tests have failed.

   The test inputs, actual outputs obtained from their execution, and the expected outputs are passed on to the testing oracle. The oracle is the person, program, or process used to determine if a test has failed.

4. Pass the failures on for debugging.

   The purpose of debugging is to determine the faults i nthe code that are actually responsible for the failures observed during testing.

Common oracles:

Common oracles:

• the “eyeball” oracle

Common oracles:

• the “eyeball” oracle

  • human inspection of output

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

• “head to head” oracles

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

• “head to head” oracles

  • comparison against outputs of an existing system

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

• “head to head” oracles

  • comparison against outputs of an existing system

• automated oracles

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

• “head to head” oracles

  • comparison against outputs of an existing system

• automated oracles

  • an upcoming lesson

Common oracles:

• the “eyeball” oracle

  • human inspection of output

  • notoriously unreliable

  • a high degree of validation

• “head to head” oracles

  • comparison against outputs of an existing system

• automated oracles

  • an upcoming lesson

  • verification but not validation

It is used, during regression testing, to quickly rerun old tests. Regression databases can quickly grow to thousands or tens of thousands of cases or more. It becomes particularly important that we not rely on the eyeball oracle for evaluating regression tests.

• Unit Test: Tests of individual subroutines and modules,

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• System Test: Test of the entire system,

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• System Test: Test of the entire system,

  • supervised by team leaders or by V&V specialists.

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• System Test: Test of the entire system,

  • supervised by team leaders or by V&V specialists.
  • Many companies have independent teams for this purpose.

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• System Test: Test of the entire system,

  • supervised by team leaders or by V&V specialists.
  • Many companies have independent teams for this purpose.

• Regression Test: Unit/Integration/System tests that are repeated after a change has been made to the code.

2 Stages of Testing

We recognize several different stages of testing. These differ in scope (how much of the program is involved) and purpose (who conducts the testing and what information do they derive from it).

• Unit Test: Tests of individual subroutines and modules,

  • usually conducted by the programmer.

• Integration Test: Tests of “subtrees” of the total project hierarchy chart (groups of subroutines calling each other).

  • generally a team responsibility.

• System Test: Test of the entire system,

  • supervised by team leaders or by V&V specialists.
  • Many companies have independent teams for this purpose.

• Regression Test: Unit/Integration/System tests that are repeated after a change has been made to the code.

• Acceptance Test: A test conducted by the customers or their representatives to decide whether to purchase/accept a developed system.

Focusing on the differing purposes of testing, …

• Unit Test: does it work?

• Integration Test: does it work?

• System Test: does it work?

• Regression Test: has it changed?

• Acceptance Test: should we pay for it?

So, while most testing has possible outcomes “pass” or “fail”, regression testing has outcomes

• Expected pass: this test used to pass, and it still does
• Expected fail: this test used to fail, and it still does
• Unexpected pass: this test used to fail, and but not it passes

• Unexpected fail: this test used to pass, and now it fails.

  • Sadly, very common. This means that while fixing one bug, we broke something else.

3 Unit Testing

We’re going to spend a lot of time taking about unit testing this semester, so it deserves some special attention now.

By testing modules in isolation from the rest of the system

• Easier to design and run extensive tests

• Much easier to debug any failures

• Errors caught much earlier

Main challenge is how to test in isolation

• Scaffolding is any code that we write, not as part of the application, but simply to support the process of Unit and Integration testing.

• Scaffolding comes in two forms

  • Drivers

  • Stubs

• Often just a simple main program that reads values, uses them to construct ADT values, apply ADT operations and print the results

• Must match the “real” API
• May need to provide simulated output parameters and return values

4 Integration Testing

Integration testing is testing that combines several modules, but still falls short of exercising the entire program all at once.

• Integration testing usually combines a small number of modules that call upon one another.

• Integration testing can be conducted

  • bottom-up

    • relieves the need for stubs

  • or top-down

It’s worth noting that unit testing and integration testing can sometimes use some of the same test inputs (and maybe the same expected outputs), because we are testing the software in different configurations.

5 System Testing

In theory, as integration testing progresses, you eventually will integration test the entire system.

However,

• Integration tests are generally

  • conducted by the software developers
  • in their development environment.

• System tests

  • may be conducted by a separate team
  • and should be done in the production environment.