Abstract

Regression testing monitors the changes, for better or worse, of the system as we make changes to it. Regression testing generally involves large numbers of tests, often selected as a mixture of test cases originally developed as unit, integration, and system tests.

In this lesson we will look at what is required for support of regression testing.

1 Regression Frameworks

What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation

  • Technically, regression tests can come from any level of normal testing

• High degree of automation

  • Regression sets can be huge

• Allowance for both expected pass and expected failure cases

  • Regression testing is about learning what has changed, not what has passed

---

Open Possibilities

• DejaGNU

• fitnesse

• Maveryx

1.1 DejaGnu

• Posix-compliant regression test framework

• Written in expect

• Strong support for multiple configurations

---

Test Results

Standard for test frameworks.

Possible outputs for any test:

• PASS: test has succeeded

• FAIL: test has failed

• XFAIL: test has failed, but the failure was expected

• XPASS: test has passed, but was expected to fail

• UNRESOLVED: The test has produced indeterminate results and requires human review for resolution

• UNTESTED: The test was not run.

  • This is a placeholder for tests that have not yet been written.

• UNSUPPORTED: The test cannot be run because of external conditions

  • (e.g., it is OS-specific and we are testing in a different OS).

---

Sample Test

set testdata {
   {"addition" "22 + 1" "23"}
   {"multiplication" "21 * 2" "42"}
   {"division" "14 / 3" "4"}
     ⋮
}
foreach pattern $testdata {
   eval "spawn ./calc [lindex $pattern 1]"
   expect {
     -re [lindex $pattern 2] {
              pass [lindex $pattern 0] }
     default {fail [lindex $pattern 0] }
   }
}

---

Running DejaGnu

runtests --tool calcTests

runs all tests in the calcTests directory, producing output like:

 PASS addition
 PASS multiplication
   ⋮
 # of expected passes          12
 # of unexpected failures       2

1.2 fitnesse

• Tests composed on Wiki pages

  • Encourages collaborative dvelopment & readable documentation

• Tool walks pages to run tests

• example

---

Is fitnesse a Regression Framework?

Advertises itself as such.

• Not clear to me how to extend it past simple text I/O

  • Is this really a system-level approach?

• Concept of expected fail is missing

2 Maveryx

• Java & Android GUI testing

  • Traditional GUI element searching replaced by “object recognition” experts
    • Is this really different from programmed searches?
    • Features some “fuzzy” search criteria

• Scripting in Java

• Use JUnit 4 assertions to write your oracle code.

---

Is Maveryx a Regression Framework?

Advertises itself as such.

• Appears to have legitimate system-level focus

• Expected fail?