System Testing

Steven J Zeil

Last modified: Oct 11, 2016
Contents:
1 Integration and system testing
1.1 Unit to Integration
1.2 Integration to System
2 Test Coverage
2.1 Coverage Measures
2.2 C/C++ - gcov
2.3 Java
3 Oracles
3.1 expect
3.2 *Unit
3.3 Testing GUI systems
3.4 Web systems
3.5 Selenium

1 Integration and system testing

1.1 Unit to Integration

1.2 Integration to System

2 Test Coverage

Although we can monitor test coverage during unit test, it’s more common to do this during integration and system test.

2.1 Coverage Measures

We have previously reviewed:

2.2 C/C++ - gcov

Monitoring Statement Coverage with gcov

As an example, look at testing the three search functions in

arrayUtils.h
#ifndef ARRAYUTILS_H
#define ARRAYUTILS_H



//  Add to the end
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter
template <typename T>
void addToEnd (T* array, int& size, T value)
{
   array[size] = value;
   ++size;
}



// Add value into array[index], shifting all elements already in positions
//    index..size-1 up one, to make room.
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter

template <typename T>
void addElement (T* array, int& size, int index, T value)
{
  // Make room for the insertion
  int toBeMoved = size - 1;
  while (toBeMoved >= index) {
    array[toBeMoved+1] = array[toBeMoved];
    --toBeMoved;
  }
  // Insert the new value
  array[index] = value;
  ++size;
}


// Assume the elements of the array are already in order
// Find the position where value could be added to keep
//    everything in order, and insert it there.
// Return the position where it was inserted
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter

template <typename T>
int addInOrder (T* array, int& size, T value)
{
  // Make room for the insertion
  int toBeMoved = size - 1;
  while (toBeMoved >= 0 && value < array[toBeMoved]) {
    array[toBeMoved+1] = array[toBeMoved];
    --toBeMoved;
  }
  // Insert the new value
  array[toBeMoved+1] = value;
  ++size;
  return toBeMoved+1;
}


// Search an array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
    int loc;

    for (loc = 0; loc < listLength; loc++)
        if (list[loc] == searchItem)
            return loc;

    return -1;
}


// Search an ordered array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int seqOrderedSearch(const T list[], int listLength, T searchItem)
{
    int loc = 0;

    while (loc < listLength && list[loc] < searchItem)
      {
       ++loc;
      }
    if (loc < listLength && list[loc] == searchItem)
       return loc;
    else
       return -1;
}


// Removes an element from the indicated position in the array, moving
// all elements in higher positions down one to fill in the gap.
template <typename T>
void removeElement (T* array, int& size, int index)
{
  int toBeMoved = index + 1;
  while (toBeMoved < size) {
    array[toBeMoved] = array[toBeMoved+1];
    ++toBeMoved;
  }
  --size;
}



// Search an ordered array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int binarySearch(const T list[], int listLength, T searchItem)
{
    int first = 0;
    int last = listLength - 1;
    int mid;

    bool found = false;

    while (first <= last && !found)
    {
        mid = (first + last) / 2;

        if (list[mid] == searchItem)
            found = true;
        else 
            if (searchItem < list[mid])
                last = mid - 1;
            else
                first = mid + 1;
    }

    if (found) 
        return mid;
    else
        return -1;
}





#endif

with test driver

gcovDemo.cpp
#include <cassert>
#include <iostream>
#include <sstream>
#include <string>

#include "arrayUtils.h"

using namespace std;



// Unit test driver for array search functions





int main(int argc, char** argv)
{
  // Repeatedly reads tests from cin
  // Each test consists of a line containing one or more words. 
  // The first word is one that we want to search for. The
  // remaining words are placed into an array and represent the collection
  // we will search through.

  string line;
  getline (cin, line);
  while (cin)
    {
      istringstream in (line);
      cout << line << endl;
      string toSearchFor;
      in >> toSearchFor;
      int nWords = 0;
      string words[100];
      while (in >> words[nWords])
	++nWords;
      
      cout << seqSearch (words, nWords, toSearchFor)
	   << " "
	   << seqOrderedSearch (words, nWords, toSearchFor)
	   << " "
	   << binarySearch (words, nWords, toSearchFor)
	   << endl;

      getline (cin, line);
    }

  return 0;
}

which reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.


Compiling for gcov Statement Coverage


Running Tests with gcov


Viewing Your Report


Sample Statement Coverage Report

     -:   69:template <typename T>
     -:   70:int seqSearch(const T list[], int listLength, T searchItem)
     -:   71:{
     1:   72:    int loc;
     -:   73:
     2:   74:    for (loc = 0; loc < listLength; loc++)
     2:   75:        if (list[loc] == searchItem)
     1:   76:            return loc;
     -:   77:
 #####:   78:    return -1;
     -:   79:}


Monitoring Branch Coverage with gcov

gcov can report on branches taken.


Reading gcov Branch Info


But What is a “Branch”?


Example: gcov Branch Coverage report

        -:   84:template <typename T>
        -:   85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
        -:   86:{
        1:   87:    int loc = 0;
        -:   88:
        1:   89:    while (loc < listLength && list[loc] < searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
branch  3 taken 1
        -:   90:      {
    #####:   91:       ++loc;
branch  0 never executed
        -:   92:      }
        1:   93:    if (loc < listLength && list[loc] == searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
        1:   94:       return loc;
branch  0 taken 1
        -:   95:    else
    #####:   96:       return -1;
        -:   97:}

2.3 Java

Java Coverage Tools


Clover


JaCoCo

Java Code Coverage


Example: JaCoCo in Eclipse

Once you have the plugin installed,

  1. Open any Eclipse Java project.
  2. Right-click on a unit test or executable. Look for “Coverage As
  3. After execution has completed coverage results are shown

 


Example: JaCoCo in Ant

Working with our Code Annotation project, add a dependency on the JaCoCo library:

jacoco-ivy.xml.listing
<ivy-module version="2.0">
  <info organisation="edu.odu.cs" module="codeAnnotation" revision="1.0"/>
  <publications>
    <artifact name="codeAnnotation" type="jar" ext="jar"/>
    <artifact name="codeAnnotation-src" type="source" ext="zip"/>
  </publications>
  <dependencies>
    <dependency org="de.jflex" name="jflex" rev="1.4.3"/>
    <dependency org="junit" name="junit" rev="4.10"/>
    <dependency org="org.jacoco" name="org.jacoco.ant" 
       rev="latest.integration"/>
  </dependencies>
</ivy-module>

Example: JaCoCo in Ant (cont.)

jacoco-build.xml.listing
<project name="codeAnnotation" basedir="." default="build"
	 xmlns:ivy="antlib:org.apache.ivy.ant"
	 xmlns:jacoco="antlib:org.jacoco.ant"
>

  <record name="ant.log" action="start" append="false" />
	
  <path id="testCompilationPath">
    <fileset dir="lib" includes="*.jar"/>
    <pathelement path="target/classes"/>
  </path>

  <path id="testExecutionPath">
    <fileset dir="lib" includes="*.jar"/>
    <pathelement path="target/classes"/>
    <pathelement path="target/test-classes"/>
  </path>


  <property name="ivy.install.version" value="2.3.0"/>
  <property name="jsch.install.version" value="0.1.49"/>
  <property name="ivy.jar.dir" value="${basedir}/ivy"/>
  <property name="ivy.jar.file" value="${ivy.jar.dir}/ivy.jar"/>
  <property name="jsch.jar.file" value="${ivy.jar.dir}/jsch.jar"/>
  <property name="build.dir" value="build"/>
  <property name="src.dir" value="src"/>

  <target name="download-ivy" unless="skip.download">
    <mkdir dir="${ivy.jar.dir}"/>
    <echo message="installing ivy..."/>
    <get src="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar" 
	 dest="${ivy.jar.file}" usetimestamp="true"/>
  </target>

  <target name="download-jsch" unless="skip.download">
    <mkdir dir="${ivy.jar.dir}"/>
    <echo message="installing jsch..."/>
    <get src="http://repo1.maven.org/maven2/com/jcraft/jsch/${jsch.install.version}/jsch-${jsch.install.version}.jar" 
	 dest="${jsch.jar.file}" usetimestamp="true"/>
  </target>
	
  <target name="install-jsch" depends="download-jsch">
  </target>
	
  <target name="install-ivy" depends="download-ivy" 
	  description="--> install ivy">
    <path id="ivy.lib.path">
      <fileset dir="${ivy.jar.dir}" includes="*.jar"/>
    </path>
    <taskdef resource="org/apache/ivy/ant/antlib.xml" 
	     uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
  </target>

  <target name="resolve-ivy" depends="install-ivy,install-jsch" description="Resolve library dependencies">
    <ivy:retrieve/>
    <echo>ivy.default.ivy.user.dir is ${ivy.default.ivy.user.dir}</echo>
    <taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"> ➀
        <classpath refid="testExecutionPath"/>
    </taskdef>
    <taskdef classname="JFlex.anttask.JFlexTask" name="jflex">
        <classpath refid="testExecutionPath"/>
    </taskdef>
  </target>

  <target name="generateSource" depends="resolve-ivy">
    <mkdir dir="target/gen/java"/>
    <jflex file="src/main/jflex/code2html.flex"
	   destdir="target/gen/java"/>
    <jflex file="src/main/jflex/code2tex.flex"
	   destdir="target/gen/java"/>
    <jflex file="src/main/jflex/list2html.flex"
	   destdir="target/gen/java"/>
    <jflex file="src/main/jflex/list2tex.flex"
	   destdir="target/gen/java"/>
  </target>
  

  <target name="compile" depends="generateSource">   ➁
    <mkdir dir="target/classes"/>
    <javac srcdir="target/gen/java" destdir="target/classes" debug="true"
	   source="1.6" includeantruntime="false"/>
    <javac srcdir="src/main/java" destdir="target/classes" debug="true"
	   source="1.6" includeantruntime="false"/>
  </target>

  <target name="compile-tests" depends="compile">
    <mkdir dir="target/test-classes"/>
    <javac srcdir="src/test/java" destdir="target/test-classes" debug="true"
	   source="1.6" includeantruntime="false">
      <classpath refid="testCompilationPath"/>
    </javac>
  </target>

  <target name="test" depends="compile-tests">
    <mkdir dir="target/test-results/details"/>
    <jacoco:coverage destfile="target/jacoco.exec">    ➂
      <junit printsummary="yes" 
	     haltonfailure="yes" fork="yes"
	     >
	<classpath refid="testExecutionPath"/>
	<formatter type="xml"/>
	<batchtest todir="target/test-results/details">
	  <fileset dir="target/test-classes">
	    <include name="**/*Test*.class"/>
	  </fileset>
	</batchtest>
      </junit>
    </jacoco:coverage>
    <junitreport todir="target/test-results">
      <fileset dir="target/test-results/details">
        <include name="TEST-*.xml"/>
      </fileset>
      <report format="frames" todir="target/test-results/html"/>
    </junitreport>
  </target>

  <target name="coverageReport" depends="test">
    <jacoco:report>                ➃
        <executiondata>
            <file file="target/jacoco.exec"/>  ➄
        </executiondata>
                           
	<structure name="Code Annotation Project"> ➅
	  <classfiles>
	    <fileset dir="target/classes"/>
	    <fileset dir="target/test-classes"/>
	  </classfiles>
	  <sourcefiles encoding="UTF-8">
	    <fileset dir="src/main/java"/>
	    <fileset dir="src/test/java"/>
	    <fileset dir="target/gen/java"/>
	  </sourcefiles>
        </structure>
                           
        <html destdir="target/coverageReport"/>   ➆
                           
    </jacoco:report>
  </target>

  <target name="build"  depends="coverageReport">
  	<jar destfile="codeAnnotation.jar" basedir="target/classes">
  	    <manifest>
  		    <attribute name="Main-Class"
  		               value="edu.odu.cs.code2html.Code2HTML"/>
  		</manifest>
  	</jar>
  	<zip destfile="target/codeAnnotation-src.zip">
	  <fileset dir=".">
	    <include name="*.xml"/>
	    <include name="test.cpp"/>
	    <include name="*.css.cpp"/>
	    <include name="src/**/*"/>
	    <exclude name="**/*~"/>
	    <exclude name="target/**/*"/>
	  </fileset>
  	</zip>
  </target>

  <target name="publish" depends="build">
    <ivy:retrieve/>
    <ivy:publish resolver="Forge350Publish"
		 status="release"
		 update="true"
		 overwrite="true"
		 publishivy="true">
      <artifacts pattern="[artifact].[ext]"/>
    </ivy:publish>
  </target>

  <target name="clean">
    <delete dir="target"/>
  </target>

  <target name="cleaner" depends="clean">
    <delete dir="lib"/>
  </target>

  <target name="cleanest" depends="cleaner">
    <delete dir="ivy"/>
  </target>
  
</project>

Example: JaCoCo Report

3 Oracles

A testing oracle is the process, person, and/or program that determines if test output is correct

3.1 expect

Covered previously, expect is a shell for testing interactive programs.

3.2 *Unit

Can we use *Unit-style frameworks as oracles at the system test level?

3.3 Testing GUI systems


Some Open Alternatives


Marathon

For Java GUIs

def test

   $java_recorded_version = "1.6.0_24"

   with_window("Simple Widgets") {
       select("First Name", "Jalian Systems")
       select("Password", "Secret")
       assert_p("First Name", "Text", "Jalian Systems")
    }
end


Jemmy

Also for Java GUIs

3.4 Web systems


Some Open Alternatives

3.5 Selenium


Selenium Scripting


Selenese

A typical scripting statement has the form

command parameter1 [parameter2]

Parameters can be


A Sample Selenium Script

<table>
  <tr><td>open</td><td>http://mySite.com/downloads/</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Downloads</td></tr>
  <tr><td>verifyText</td><td>//h2</td><td>Terms and Conditions</td></tr>
  <tr><td>clickAndWait</td><td>//input[@value="I agree"]</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Product Selection</td></tr>
</table>

That’s right – it’s an HTML table:

open http://mySite.com/downloads/
assertTitle Downloads
verifyText //h2 Terms and Conditions
clickAndWait //input[@value=“I agree”]
assertTitle Product Selection

A Selenium “test suite” is a web page with a table of links to web pages with test cases.


Selenium Webdriver

Provides APIs to a variety of languages allowing for very similar capabilities:

Select select = new Select(driver.findElement(
       By.tagName("select")));
select.deselectAll();
select.selectByVisibleText("Edam");


Waiting

WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (
  new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(
            By.id("myDynamicElement")));

Waits up to 10 seconds for an expected element to load