What Makes a Testing Framework a Regression Framework?

Little agreement, but IMNSHO

• System-level orientation
  • Technically, regression tests can come from any level of normal testing
• High degree of automation
  • Regression sets can be huge
• Allowance for both expected pass and expected failure cases
  • Regression testing is about learning what has changed, not what has passed

1. Regression Frameworks

Open Possibilities

• DejaGNU

• fitnesse

• Maveryx

1.1 DejaGnu

• Posix-compliant regression test framework

• Written in expect

• Strong support for multiple configurations

Test Results

Standard for test frameworks.

Possible outputs for any test:

PASS : test has succeeded

FAIL : test has failed

XFAIL : test has failed, but the failure was expected

XPASS : test has passed, but was expected to fail

UNRESOLVED : The test has produced indeterminate results and requires human review for resolution

UNTESTED : The test was not run.

This is a placeholder for tests
that have not yet been written.

UNSUPPORTED : The test cannot be run because of external conditions

(e.g., it is OS-specific and we are testing in a different OS).

Sample Test

set testdata {
   {"addition" "22 + 1" "23"}
   {"multiplication" "21 * 2" "42"}
   {"division" "14 / 3" "4"}
     ⋮
}
foreach pattern $testdata {
   eval "spawn ./calc [lindex $pattern 1]"
   expect {
     -re [lindex $pattern 2] { 
              pass [lindex $pattern 0] }
     default {fail [lindex $pattern 0] }
   }
}

Running DejaGnu

runtests --tool calcTests

runs all tests in the calcTests directory, producing output like:

 PASS addition
 PASS multiplication
   ⋮
 # of expected passes          12
 # of unexpected failures       2

1.2 fitnesse

• Tests composed on Wiki pages
  • Encourages collaborative dvelopment & readable documentation

• Tool walks pages to run tests

• example

Is fitnesse a Regression Framework?

Advertises itself as such.

• Not clear to me how to extend it past simple text I/O
  • Is this really a system-level approach?

• Concept of expected fail is missing

2. Maveryx

• Java GUI testing
  • Traditional GUI element searching replaced by “object recognition” experts
    • Is this really different from programmed searches?
    • Features some “fuzzy” search criteria

• Scripting in Java

• Not clear to me how oracles handled
  • Some suggestsion that key features are slated “in the future”

Is Maveryx a Regression Framework?

Advertises itself as such.

• Appears to have legitimate system-level focus

• Expected fail?

3. Continuous Integration

When we combine

• Regression suites
  • with automated oracles
    • even if the regression suite contains mainly unit tests
• Centralized version control
  • or distributed VC with a central “official” repository
• Automated builds, capable of running tests, running analysis tools, and publishing the results on a project web site

we can rebuild and retest as developers check in changes.

3.1 Key Ideas

• Version control with a clearly identified main branch

• Automated build

• Developers commit frequently (maybe many times per day)
  • Every commit is built (for early detection of integration problems)
• Test in a clone of the production environment
  • May differ from development environments
  • Probably not checked frequently under nonrmal practice

• Make the results highly visible

Advantages

• Integration problems caught early and fixed fast
  • avoids “integration hell” []
• Immediate testing of all changes
• Emphasis on frequent check-ins encourages modularity
• Visible code quality metrics motivate developers.

Disadvantages

• Initial setup effort to set up
• Level of sophistication required of team to put build, configuration mgmt, testing, reporting, into an automated build

3.2 Software

• Lots of packages emerging for this
• Jenkins on our forge

3.3 Related ideas

• Continuous deployment publishes snapshots of deliverables as changes are checked in.

• Build light indicators
  • Not sure what else to say about this!