1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

• High degree of automation

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

• High degree of automation

  • Regression sets can be huge

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

• High degree of automation

  • Regression sets can be huge

• Allowance for both expected pass and expected failure cases

Open Possibilities

• DejaGNU

• fitnesse

• Maveryx

1.1 DejaGnu

• Posix-compliant regression test framework

• Written in expect

• Strong support for multiple configurations

Test Results

Standard for test frameworks.

Possible outputs for any test:

• PASS: test has succeeded

• FAIL: test has failed

• XFAIL: test has failed, but the failure was expected

• XPASS: test has passed, but was expected to fail

• UNRESOLVED: The test has produced indeterminate results and requires human review for resolution

• UNTESTED: The test was not run.

  • This is a placeholder for tests that have not yet been written.

• UNSUPPORTED: The test cannot be run because of external conditions

  • (e.g., it is OS-specific and we are testing in a different OS).

Sample Test

set testdata {
   {"addition" "22 + 1" "23"}
   {"multiplication" "21 * 2" "42"}
   {"division" "14 / 3" "4"}
     ⋮
}
foreach pattern $testdata {
   eval "spawn ./calc [lindex $pattern 1]"
   expect {
     -re [lindex $pattern 2] {
              pass [lindex $pattern 0] }
     default {fail [lindex $pattern 0] }
   }
}

Running DejaGnu

runtests --tool calcTests

runs all tests in the calcTests directory, producing output like:

 PASS addition
 PASS multiplication
   ⋮
 # of expected passes          12
 # of unexpected failures       2

There are lots of lists of “most popular regression testing tools” on the web. Example

• As far as I can tell, they are all system testing tools. E.g., None have the notion of expected failure
• Some of the ones almost universally listed, e.g., Selenium, explicitly deny being regression test tools on their own website.

Apparently, so few organizations actually do regression testing that people have forgotten what it means!