Abstract
Regression testing monitors the changes, for better or worse, of the system as we make changes to it. Regression testing generally involves large numbers of tests, often selected as a mixture of test cases originally developed as unit, integration, and system tests.
In this lesson we will look at what is required for support of regression testing.
What Makes a Testing Framework a Regression Framework?
Little agreement, but IMNSHO
System-level orientation
Technically, regression tests can come from any level of normal testing
High degree of automation
Allowance for both expected pass and expected failure cases
Open Possibilities
Posix-compliant regression test framework
Written in expect
Strong support for multiple configurations
Test Results
Standard for test frameworks.
Possible outputs for any test:
PASS: test has succeeded
FAIL: test has failed
XFAIL: test has failed, but the failure was expected
XPASS: test has passed, but was expected to fail
UNRESOLVED: The test has produced indeterminate results and requires human review for resolution
UNTESTED: The test was not run.
This is a placeholder for tests that have not yet been written.
UNSUPPORTED: The test cannot be run because of external conditions
(e.g., it is OS-specific and we are testing in a different OS).
Sample Test
set testdata {
{"addition" "22 + 1" "23"}
{"multiplication" "21 * 2" "42"}
{"division" "14 / 3" "4"}
⋮
}
foreach pattern $testdata {
eval "spawn ./calc [lindex $pattern 1]"
expect {
-re [lindex $pattern 2] {
pass [lindex $pattern 0] }
default {fail [lindex $pattern 0] }
}
}
Running DejaGnu
runtests --tool calcTests
runs all tests in the calcTests directory, producing output like:
PASS addition
PASS multiplication
⋮
# of expected passes 12
# of unexpected failures 2
Tests composed on Wiki pages
Tool walks pages to run tests
Is fitnesse a Regression Framework?
Advertises itself as such.
Not clear to me how to extend it past simple text I/O
Concept of expected fail is missing
Java & Android GUI testing
Scripting in Java
Use JUnit 4 assertions to write your oracle code.
Is Maveryx a Regression Framework?
Advertises itself as such.
Appears to have legitimate system-level focus
Expected fail?