Abstract

Regression testing monitors the changes, for better or worse, of the system as we make changes to it. Regression testing generally involves large numbers of tests, often selected as a mixture of test cases originally developed as unit, integration, and system tests.

In this lesson we will look at what is required for support of regression testing.

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

• High degree of automation

  • Regression sets can be huge

• Allowance for both expected pass and expected failure cases

  • Regression testing is about learning what has changed, not what has passed

---

Open Possibilities

• DejaGNU

• fitnesse

• Maveryx

1.1 DejaGnu

• Posix-compliant regression test framework

• Written in expect

• Strong support for multiple configurations

---

Test Results

Standard for test frameworks.

Possible outputs for any test:

• PASS: test has succeeded

• FAIL: test has failed

• XFAIL: test has failed, but the failure was expected

• XPASS: test has passed, but was expected to fail

• UNRESOLVED: The test has produced indeterminate results and requires human review for resolution

• UNTESTED: The test was not run.

  • This is a placeholder for tests that have not yet been written.

• UNSUPPORTED: The test cannot be run because of external conditions

  • (e.g., it is OS-specific and we are testing in a different OS).

---

Sample Test

set testdata {
   {"addition" "22 + 1" "23"}
   {"multiplication" "21 * 2" "42"}
   {"division" "14 / 3" "4"}
     ⋮
}
foreach pattern $testdata {
   eval "spawn ./calc [lindex $pattern 1]"
   expect {
     -re [lindex $pattern 2] {
              pass [lindex $pattern 0] }
     default {fail [lindex $pattern 0] }
   }
}

---

Running DejaGnu

runtests --tool calcTests

runs all tests in the calcTests directory, producing output like:

 PASS addition
 PASS multiplication
   ⋮
 # of expected passes          12
 # of unexpected failures       2

1.2 What Else?

There are lots of lists of “most popular regression testing tools” on the web. Example

• As far as I can tell, they are all system testing tools. E.g., None have the notion of expected failure
• Some of the ones almost universally listed, e.g., Selenium, explicitly deny being regression test tools on their own website.

Apparently, so few organizations actually do regression testing that people have forgotten what it means!