Start with your unit tests.
Supplement with tests of “interesting” interactions among modules.
System testing is the limiting case of integration testing.
Works with entire program’s inputs and outputs
A challenge when inputs and outputs involve GUIs, databases, and other non-text, non-API components.
Generally fewer tests than under unit and integration testing.
But may run a lot longer
Although we can monitor test coverage during unit test, it’s more common to do this during integration and system test.
During Unit test, we are working with a lot of “fake” code (drivers and stubs/mocks).
We certainly don’t care how well our drivers and stubs were covered!
Integration and system testing gets to more realistic ’combinations" of operations.
We have previously reviewed:
Black-Box Testing
White-Box Testing
Structural Testing (a.k.a., “path testing”
Mutation testing
Monitoring Statement Coverage with gcov
As an example, look at testing the three search functions in
#ifndef ARRAYUTILS_H
#define ARRAYUTILS_H
// Add to the end
// - Assumes that we have a separate integer (size) indicating how
// many elements are in the array
// - and that the "true" size of the array is at least one larger
// than the current value of that counter
template <typename T>
void addToEnd (T* array, int& size, T value)
{
array[size] = value;
++size;
}
// Add value into array[index], shifting all elements already in positions
// index..size-1 up one, to make room.
// - Assumes that we have a separate integer (size) indicating how
// many elements are in the array
// - and that the "true" size of the array is at least one larger
// than the current value of that counter
template <typename T>
void addElement (T* array, int& size, int index, T value)
{
// Make room for the insertion
int toBeMoved = size - 1;
while (toBeMoved >= index) {
array[toBeMoved+1] = array[toBeMoved];
--toBeMoved;
}
// Insert the new value
array[index] = value;
++size;
}
// Assume the elements of the array are already in order
// Find the position where value could be added to keep
// everything in order, and insert it there.
// Return the position where it was inserted
// - Assumes that we have a separate integer (size) indicating how
// many elements are in the array
// - and that the "true" size of the array is at least one larger
// than the current value of that counter
template <typename T>
int addInOrder (T* array, int& size, T value)
{
// Make room for the insertion
int toBeMoved = size - 1;
while (toBeMoved >= 0 && value < array[toBeMoved]) {
array[toBeMoved+1] = array[toBeMoved];
--toBeMoved;
}
// Insert the new value
array[toBeMoved+1] = value;
++size;
return toBeMoved+1;
}
// Search an array for a given value, returning the index where
// found or -1 if not found.
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
int loc;
for (loc = 0; loc < listLength; loc++)
if (list[loc] == searchItem)
return loc;
return -1;
}
// Search an ordered array for a given value, returning the index where
// found or -1 if not found.
template <typename T>
int seqOrderedSearch(const T list[], int listLength, T searchItem)
{
int loc = 0;
while (loc < listLength && list[loc] < searchItem)
{
++loc;
}
if (loc < listLength && list[loc] == searchItem)
return loc;
else
return -1;
}
// Removes an element from the indicated position in the array, moving
// all elements in higher positions down one to fill in the gap.
template <typename T>
void removeElement (T* array, int& size, int index)
{
int toBeMoved = index + 1;
while (toBeMoved < size) {
array[toBeMoved] = array[toBeMoved+1];
++toBeMoved;
}
--size;
}
// Search an ordered array for a given value, returning the index where
// found or -1 if not found.
template <typename T>
int binarySearch(const T list[], int listLength, T searchItem)
{
int first = 0;
int last = listLength - 1;
int mid;
bool found = false;
while (first <= last && !found)
{
mid = (first + last) / 2;
if (list[mid] == searchItem)
found = true;
else
if (searchItem < list[mid])
last = mid - 1;
else
first = mid + 1;
}
if (found)
return mid;
else
return -1;
}
#endif
with test driver
#include <cassert>
#include <iostream>
#include <sstream>
#include <string>
#include "arrayUtils.h"
using namespace std;
// Unit test driver for array search functions
int main(int argc, char** argv)
{
// Repeatedly reads tests from cin
// Each test consists of a line containing one or more words.
// The first word is one that we want to search for. The
// remaining words are placed into an array and represent the collection
// we will search through.
string line;
getline (cin, line);
while (cin)
{
istringstream in (line);
cout << line << endl;
string toSearchFor;
in >> toSearchFor;
int nWords = 0;
string words[100];
while (in >> words[nWords])
++nWords;
cout << seqSearch (words, nWords, toSearchFor)
<< " "
<< seqOrderedSearch (words, nWords, toSearchFor)
<< " "
<< binarySearch (words, nWords, toSearchFor)
<< endl;
getline (cin, line);
}
return 0;
}
which reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.
Compiling for gcov Statement Coverage
To use gcov, we compile with special options
-fprofile-arcs -ftest-coverage
When the code has been compiled, in addition to the usual files there will be several files with endings like .gcno
These hold data on where the statements and branches in our code are.
Running Tests with gcov
Run your tests normally.
As you test, a *.gcda
file will accumulate data on your test coverage.
Viewing Your Report
gcov
_mainProgram_
Sample Statement Coverage Report
-: 69:template <typename T>
-: 70:int seqSearch(const T list[], int listLength, T searchItem)
-: 71:{
1: 72: int loc;
-: 73:
2: 74: for (loc = 0; loc < listLength; loc++)
2: 75: if (list[loc] == searchItem)
1: 76: return loc;
-: 77:
#####: 78: return -1;
-: 79:}
Monitoring Branch Coverage with gcov
gcov can report on branches taken.
gcov
command:
gcov -b -c
_mainProgram_Reading gcov Branch Info
and number of times each branch was taken
But What is a “Branch”?
A “branch” is anything that causes the code to not continue on in straight-line fashion
&&
and ||
operators introduce their own branchesIn practice, this can be very hard to interpret
Example: gcov Branch Coverage report
-: 84:template <typename T>
-: 85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
-: 86:{
1: 87: int loc = 0;
-: 88:
1: 89: while (loc < listLength && list[loc] < searchItem)
branch 0 taken 0
call 1 returns 1
branch 2 taken 0
branch 3 taken 1
-: 90: {
#####: 91: ++loc;
branch 0 never executed
-: 92: }
1: 93: if (loc < listLength && list[loc] == searchItem)
branch 0 taken 0
call 1 returns 1
branch 2 taken 0
1: 94: return loc;
branch 0 taken 1
-: 95: else
#####: 96: return -1;
-: 97:}
Report is organized by basic blocks, straight-line sequences of code terminated by a branch or a call
Hard to map to specific source code constructs
Java Coverage Tools
Clover
Commercial product, currently free for open-source projects
Works in “traditional” coverage tool fashion
Test optimization: can re-run only those tests that covered changed code
JaCoCo
Java Code Coverage
line and branch coverage
Instrumentation is done on the fly
Works with Eclipse
Works with Maven & Ant
<java>
and <junit>
tasks inside a <jacoco:coverage>
elementExample: JaCoCo in Eclipse
Once you have the plugin installed,
Coverage As
”
Function of this is the same as “Run As
” and “Debug As
”, but monitors coverage during execution.
After execution has completed coverage results are shown
Example: JaCoCo in Ant
Working with our Code Annotation project, add a dependency on the JaCoCo library:
<ivy-module version="2.0">
<info organisation="edu.odu.cs" module="codeAnnotation" revision="1.0"/>
<publications>
<artifact name="codeAnnotation" type="jar" ext="jar"/>
<artifact name="codeAnnotation-src" type="source" ext="zip"/>
</publications>
<dependencies>
<dependency org="de.jflex" name="jflex" rev="1.4.3"/>
<dependency org="junit" name="junit" rev="4.10"/>
<dependency org="org.jacoco" name="org.jacoco.ant"
rev="latest.integration"/>
</dependencies>
</ivy-module>
Example: JaCoCo in Ant (cont.)
<project name="codeAnnotation" basedir="." default="build"
xmlns:ivy="antlib:org.apache.ivy.ant"
xmlns:jacoco="antlib:org.jacoco.ant"
>
<record name="ant.log" action="start" append="false" />
<path id="testCompilationPath">
<fileset dir="lib" includes="*.jar"/>
<pathelement path="target/classes"/>
</path>
<path id="testExecutionPath">
<fileset dir="lib" includes="*.jar"/>
<pathelement path="target/classes"/>
<pathelement path="target/test-classes"/>
</path>
<property name="ivy.install.version" value="2.3.0"/>
<property name="jsch.install.version" value="0.1.49"/>
<property name="ivy.jar.dir" value="${basedir}/ivy"/>
<property name="ivy.jar.file" value="${ivy.jar.dir}/ivy.jar"/>
<property name="jsch.jar.file" value="${ivy.jar.dir}/jsch.jar"/>
<property name="build.dir" value="build"/>
<property name="src.dir" value="src"/>
<target name="download-ivy" unless="skip.download">
<mkdir dir="${ivy.jar.dir}"/>
<echo message="installing ivy..."/>
<get src="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar"
dest="${ivy.jar.file}" usetimestamp="true"/>
</target>
<target name="download-jsch" unless="skip.download">
<mkdir dir="${ivy.jar.dir}"/>
<echo message="installing jsch..."/>
<get src="http://repo1.maven.org/maven2/com/jcraft/jsch/${jsch.install.version}/jsch-${jsch.install.version}.jar"
dest="${jsch.jar.file}" usetimestamp="true"/>
</target>
<target name="install-jsch" depends="download-jsch">
</target>
<target name="install-ivy" depends="download-ivy"
description="--> install ivy">
<path id="ivy.lib.path">
<fileset dir="${ivy.jar.dir}" includes="*.jar"/>
</path>
<taskdef resource="org/apache/ivy/ant/antlib.xml"
uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
</target>
<target name="resolve-ivy" depends="install-ivy,install-jsch" description="Resolve library dependencies">
<ivy:retrieve/>
<echo>ivy.default.ivy.user.dir is ${ivy.default.ivy.user.dir}</echo>
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"> ➀
<classpath refid="testExecutionPath"/>
</taskdef>
<taskdef classname="JFlex.anttask.JFlexTask" name="jflex">
<classpath refid="testExecutionPath"/>
</taskdef>
</target>
<target name="generateSource" depends="resolve-ivy">
<mkdir dir="target/gen/java"/>
<jflex file="src/main/jflex/code2html.flex"
destdir="target/gen/java"/>
<jflex file="src/main/jflex/code2tex.flex"
destdir="target/gen/java"/>
<jflex file="src/main/jflex/list2html.flex"
destdir="target/gen/java"/>
<jflex file="src/main/jflex/list2tex.flex"
destdir="target/gen/java"/>
</target>
<target name="compile" depends="generateSource"> ➁
<mkdir dir="target/classes"/>
<javac srcdir="target/gen/java" destdir="target/classes" debug="true"
source="1.6" includeantruntime="false"/>
<javac srcdir="src/main/java" destdir="target/classes" debug="true"
source="1.6" includeantruntime="false"/>
</target>
<target name="compile-tests" depends="compile">
<mkdir dir="target/test-classes"/>
<javac srcdir="src/test/java" destdir="target/test-classes" debug="true"
source="1.6" includeantruntime="false">
<classpath refid="testCompilationPath"/>
</javac>
</target>
<target name="test" depends="compile-tests">
<mkdir dir="target/test-results/details"/>
<jacoco:coverage destfile="target/jacoco.exec"> ➂
<junit printsummary="yes"
haltonfailure="yes" fork="yes"
>
<classpath refid="testExecutionPath"/>
<formatter type="xml"/>
<batchtest todir="target/test-results/details">
<fileset dir="target/test-classes">
<include name="**/*Test*.class"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<junitreport todir="target/test-results">
<fileset dir="target/test-results/details">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="target/test-results/html"/>
</junitreport>
</target>
<target name="coverageReport" depends="test">
<jacoco:report> ➃
<executiondata>
<file file="target/jacoco.exec"/> ➄
</executiondata>
<structure name="Code Annotation Project"> ➅
<classfiles>
<fileset dir="target/classes"/>
<fileset dir="target/test-classes"/>
</classfiles>
<sourcefiles encoding="UTF-8">
<fileset dir="src/main/java"/>
<fileset dir="src/test/java"/>
<fileset dir="target/gen/java"/>
</sourcefiles>
</structure>
<html destdir="target/coverageReport"/> ➆
</jacoco:report>
</target>
<target name="build" depends="coverageReport">
<jar destfile="codeAnnotation.jar" basedir="target/classes">
<manifest>
<attribute name="Main-Class"
value="edu.odu.cs.code2html.Code2HTML"/>
</manifest>
</jar>
<zip destfile="target/codeAnnotation-src.zip">
<fileset dir=".">
<include name="*.xml"/>
<include name="test.cpp"/>
<include name="*.css.cpp"/>
<include name="src/**/*"/>
<exclude name="**/*~"/>
<exclude name="target/**/*"/>
</fileset>
</zip>
</target>
<target name="publish" depends="build">
<ivy:retrieve/>
<ivy:publish resolver="Forge350Publish"
status="release"
update="true"
overwrite="true"
publishivy="true">
<artifacts pattern="[artifact].[ext]"/>
</ivy:publish>
</target>
<target name="clean">
<delete dir="target"/>
</target>
<target name="cleaner" depends="clean">
<delete dir="lib"/>
</target>
<target name="cleanest" depends="cleaner">
<delete dir="ivy"/>
</target>
</project>
➀ Once the dependencies are resolved, we can activate the JaCoCo tasks.
➁ Note that there is no change at all in compilation
➂ And minimal change to execution
fork="true"
because
➃ Preparation of reports starts here
Example: JaCoCo Report
A testing oracle is the process, person, and/or program that determines if test output is correct
Covered previously, expect is a shell for testing interactive programs.
an extension of TCL (a portable shell script).
Largely confined to text streams as input/output
Can we use *Unit-style frameworks as oracles at the system test level?
The very question is heresy to many *Unit advocates
But, why not?
Such tests do not (should not) be at the expense of having done earlier “proper” unit testing.
Particularly in Java, MyClass.main(String[]) can be called just like any other function
And System.in (cin) and System.out (cout) can be rerouted to/from files or internal strings
Major limitation is the accessibility of system inputs & outputs.
Scripting or record/playback: playing back input events for
Capture of results
Some Open Alternatives
Marathon
For Java GUIs
Recorder captures AWT/swing events as JRuby scripts
Scripts can then be edited to alter inputs, add assertions, etc.
def test
$java_recorded_version = "1.6.0_24"
with_window("Simple Widgets") {
select("First Name", "Jalian Systems")
select("Password", "Secret")
assert_p("First Name", "Text", "Jalian Systems")
}
end
Jemmy
Also for Java GUIs
Tests scripted as Java
Integrates with JUnit
Some Open Alternatives
Selenium Scripting
Actions do things to elements.
E.g., click buttons, select options
Accessors examine the application state
Assertions validate the state
Each assertion has 3 modes
Selenese
A typical scripting statement has the form
command parameter1 [parameter2]
Parameters can be
locators for finding a UI element within a page (xpath)
text patterns
variable names
A Sample Selenium Script
<table>
<tr><td>open</td><td>http://mySite.com/downloads/</td><td></td></tr>
<tr><td>assertTitle</td><td></td><td>Downloads</td></tr>
<tr><td>verifyText</td><td>//h2</td><td>Terms and Conditions</td></tr>
<tr><td>clickAndWait</td><td>//input[@value="I agree"]</td><td></td></tr>
<tr><td>assertTitle</td><td></td><td>Product Selection</td></tr>
</table>
That’s right – it’s an HTML table:
open | http://mySite.com/downloads/ | |
assertTitle | Downloads | |
verifyText | //h2 | Terms and Conditions |
clickAndWait | //input[@value=“I agree”] | |
assertTitle | Product Selection |
A Selenium “test suite” is a web page with a table of links to web pages with test cases.
Selenium Webdriver
Provides APIs to a variety of languages allowing for very similar capabilities:
Select select = new Select(driver.findElement(
By.tagName("select")));
select.deselectAll();
select.selectByVisibleText("Edam");
Waiting
WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (
new WebDriverWait(driver, 10))
.until(ExpectedConditions.presenceOfElementLocated(
By.id("myDynamicElement")));
Waits up to 10 seconds for an expected element to load