Regression Testing
Steven J Zeil
Abstract
Regression testing monitors the changes, for better or worse, of the system as we make changes to it. Regression testing generally involves large numbers of tests, often selected as a mixture of test cases originally developed as unit, integration, and system tests.
In this lesson we will look at what is required for support of regression testing.
1 Regression Frameworks
What Makes a Testing Framework a Regression Framework?
Little agreement, but IMNSHO
-
System-level orientation
-
Technically, regression tests can come from any level of normal testing
-
-
High degree of automation
- Regression sets can be huge
-
Allowance for both expected pass and expected failure cases
- Regression testing is about learning what has changed, not what has passed
Open Possibilities
1.1 DejaGnu
-
Posix-compliant regression test framework
-
Written in expect
-
Strong support for multiple configurations
Test Results
Standard for test frameworks.
Possible outputs for any test:
-
PASS: test has succeeded
-
FAIL: test has failed
-
XFAIL: test has failed, but the failure was expected
-
XPASS: test has passed, but was expected to fail
-
UNRESOLVED: The test has produced indeterminate results and requires human review for resolution
-
UNTESTED: The test was not run.
-
This is a placeholder for tests that have not yet been written.
-
-
UNSUPPORTED: The test cannot be run because of external conditions
-
(e.g., it is OS-specific and we are testing in a different OS).
-
Sample Test
set testdata {
{"addition" "22 + 1" "23"}
{"multiplication" "21 * 2" "42"}
{"division" "14 / 3" "4"}
⋮
}
foreach pattern $testdata {
eval "spawn ./calc [lindex $pattern 1]"
expect {
-re [lindex $pattern 2] {
pass [lindex $pattern 0] }
default {fail [lindex $pattern 0] }
}
}
Running DejaGnu
runtests --tool calcTests
runs all tests in the calcTests directory, producing output like:
PASS addition
PASS multiplication
⋮
# of expected passes 12
# of unexpected failures 2
1.2 fitnesse
-
Tests composed on Wiki pages
- Encourages collaborative dvelopment & readable documentation
-
Tool walks pages to run tests
Is fitnesse a Regression Framework?
Advertises itself as such.
-
Not clear to me how to extend it past simple text I/O
- Is this really a system-level approach?
-
Concept of expected fail is missing
2 Maveryx
-
Java & Android GUI testing
- Traditional GUI element searching replaced by “object recognition” experts
- Is this really different from programmed searches?
- Features some “fuzzy” search criteria
- Traditional GUI element searching replaced by “object recognition” experts
-
Scripting in Java
-
Use JUnit 4 assertions to write your oracle code.
Is Maveryx a Regression Framework?
Advertises itself as such.
-
Appears to have legitimate system-level focus
-
Expected fail?