System Testing

Steven J Zeil

Last modified: Sep 14, 2017
Contents:

Abstract

System testing adds particular challenges to testing because it is constrained to work with the actual system inputs and outputs. Compared to unit testing, we have less control over both the input supply and the capture and evaluation of the output.

In this lesson we look at tools for measuring quality of system testing. We will explore the difficulties that arise when dealing with non-text input and output, particularly graphical interfaces, and will look at some of the support available for testing at this level.

1 Integration and system testing

1.1 Unit to Integration

1.2 Integration to System

2 Testing Phases and the Build Manager

2.1 How often should we run integration/system tests?

Plausible answers:

These kinds of runs are often triggered automatically.

2.2 Separating the Tests

For Java projects, we already have a separation of code into src/main and src/test. We might consider adding src/integrationTest and src/systemTest.

Each test directory ty[ically has source code (e.g., src/itest/java) and maybe data files (e.g., src/systest/data).

2.3 Building the Tests

In Gradle, we can add test directories with the TestsSets plugin.

In build.gradle:

apply plugin: 'java'
apply plugin: 'org.unbroken-dome.test-sets'

⋮
testSets {
    integrationTest
    systemTest
} 

This adds to a Java project new tasks to compile and run tests from src/integrationTest/java and src/systemTest/java.

You can configure these tasks independently, e.g.,

integrationTest.mustRunAfter test  ➀

integrationTest {
    ignoreFailures = true          ➁
}
test {
    ignoreFailures = false         ➂
}

3 Test Coverage

Although we can monitor test coverage during unit test, it’s more common to do this during integration and system test.

3.1 Coverage Measures

We have previously reviewed:

3.2 C/C++ - gcov

Monitoring Statement Coverage with gcov

As an example, look at testing the three search functions in

arrayUtils.h
#ifndef ARRAYUTILS_H
#define ARRAYUTILS_H



//  Add to the end
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter
template <typename T>
void addToEnd (T* array, int& size, T value)
{
   array[size] = value;
   ++size;
}



// Add value into array[index], shifting all elements already in positions
//    index..size-1 up one, to make room.
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter

template <typename T>
void addElement (T* array, int& size, int index, T value)
{
  // Make room for the insertion
  int toBeMoved = size - 1;
  while (toBeMoved >= index) {
    array[toBeMoved+1] = array[toBeMoved];
    --toBeMoved;
  }
  // Insert the new value
  array[index] = value;
  ++size;
}


// Assume the elements of the array are already in order
// Find the position where value could be added to keep
//    everything in order, and insert it there.
// Return the position where it was inserted
//  - Assumes that we have a separate integer (size) indicating how
//     many elements are in the array
//  - and that the "true" size of the array is at least one larger 
//      than the current value of that counter

template <typename T>
int addInOrder (T* array, int& size, T value)
{
  // Make room for the insertion
  int toBeMoved = size - 1;
  while (toBeMoved >= 0 && value < array[toBeMoved]) {
    array[toBeMoved+1] = array[toBeMoved];
    --toBeMoved;
  }
  // Insert the new value
  array[toBeMoved+1] = value;
  ++size;
  return toBeMoved+1;
}


// Search an array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int seqSearch(const T list[], int listLength, T searchItem)
{
    int loc;

    for (loc = 0; loc < listLength; loc++)
        if (list[loc] == searchItem)
            return loc;

    return -1;
}


// Search an ordered array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int seqOrderedSearch(const T list[], int listLength, T searchItem)
{
    int loc = 0;

    while (loc < listLength && list[loc] < searchItem)
      {
       ++loc;
      }
    if (loc < listLength && list[loc] == searchItem)
       return loc;
    else
       return -1;
}


// Removes an element from the indicated position in the array, moving
// all elements in higher positions down one to fill in the gap.
template <typename T>
void removeElement (T* array, int& size, int index)
{
  int toBeMoved = index + 1;
  while (toBeMoved < size) {
    array[toBeMoved] = array[toBeMoved+1];
    ++toBeMoved;
  }
  --size;
}



// Search an ordered array for a given value, returning the index where 
//    found or -1 if not found.
template <typename T>
int binarySearch(const T list[], int listLength, T searchItem)
{
    int first = 0;
    int last = listLength - 1;
    int mid;

    bool found = false;

    while (first <= last && !found)
    {
        mid = (first + last) / 2;

        if (list[mid] == searchItem)
            found = true;
        else 
            if (searchItem < list[mid])
                last = mid - 1;
            else
                first = mid + 1;
    }

    if (found) 
        return mid;
    else
        return -1;
}





#endif

with test driver

gcovDemo.cpp
#include <cassert>
#include <iostream>
#include <sstream>
#include <string>

#include "arrayUtils.h"

using namespace std;



// Unit test driver for array search functions





int main(int argc, char** argv)
{
  // Repeatedly reads tests from cin
  // Each test consists of a line containing one or more words. 
  // The first word is one that we want to search for. The
  // remaining words are placed into an array and represent the collection
  // we will search through.

  string line;
  getline (cin, line);
  while (cin)
    {
      istringstream in (line);
      cout << line << endl;
      string toSearchFor;
      in >> toSearchFor;
      int nWords = 0;
      string words[100];
      while (in >> words[nWords])
	++nWords;
      
      cout << seqSearch (words, nWords, toSearchFor)
	   << " "
	   << seqOrderedSearch (words, nWords, toSearchFor)
	   << " "
	   << binarySearch (words, nWords, toSearchFor)
	   << endl;

      getline (cin, line);
    }

  return 0;
}

which reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.


Compiling for gcov Statement Coverage


Running Tests with gcov


Viewing Your Report


Sample Statement Coverage Report

     -:   69:template <typename T>
     -:   70:int seqSearch(const T list[], int listLength, T searchItem)
     -:   71:{
     1:   72:    int loc;
     -:   73:
     2:   74:    for (loc = 0; loc < listLength; loc++)
     2:   75:        if (list[loc] == searchItem)
     1:   76:            return loc;
     -:   77:
 #####:   78:    return -1;
     -:   79:}


Monitoring Branch Coverage with gcov

gcov can report on branches taken.


Reading gcov Branch Info


But What is a “Branch”?


Example: gcov Branch Coverage report

        -:   84:template <typename T>
        -:   85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
        -:   86:{
        1:   87:    int loc = 0;
        -:   88:
        1:   89:    while (loc < listLength && list[loc] < searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
branch  3 taken 1
        -:   90:      {
    #####:   91:       ++loc;
branch  0 never executed
        -:   92:      }
        1:   93:    if (loc < listLength && list[loc] == searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
        1:   94:       return loc;
branch  0 taken 1
        -:   95:    else
    #####:   96:       return -1;
        -:   97:}

3.3 Java

Java Coverage Tools


Clover


JaCoCo

Java Code Coverage


Example: JaCoCo in Eclipse

Using JaCoCo in Eclipse

Once you have the plugin installed,

  1. Open any Eclipse Java project.
  2. Right-click on a unit test or executable. Look for “Coverage As
    • Function of this is the same as “Run As” and “Debug As”, but monitors coverage during execution.

  3. After execution has completed coverage results are shown

    • A summary in the console area under the “Coverage” tab
    • Details in the Java editor, as color coding
      • Green means that all branches were covered
      • Red means that none were covered.
      • Yellow means that some were covered.

 


Example: JaCoCo in Gradle

In build.gradle:

apply plugin: 'java'
apply plugin: 'jacoco'
⋮
check.dependsOn jacocoTestReport

The last line is because I typically have a task named “check” that is my target for report generation. In other words, I plan to use

./gradlew check

to prepare all of my project reports, so I add a dependency between each kind of report I add and the check task.


Example: JaCoCo Report

4 Oracles

A testing oracle is the process, person, and/or program that determines if test output is correct

4.1 expect

Covered previously, expect is a shell for testing interactive programs.

4.2 *Unit

Can we use *Unit-style frameworks as oracles at the system test level?

4.3 Testing GUI systems


Some Open Alternatives


Marathon

For Java GUIs

def test

   $java_recorded_version = "1.6.0_24"

   with_window("Simple Widgets") {
       select("First Name", "Jalian Systems")
       select("Password", "Secret")
       assert_p("First Name", "Text", "Jalian Systems")
    }
end


Jemmy

Also for Java GUIs

4.4 Web systems


Some Open Alternatives

4.5 Selenium


Selenium Scripting


Selenese

A typical scripting statement has the form

command parameter1 [parameter2]

Parameters can be


A Sample Selenium Script

<table>
  <tr><td>open</td><td>http://mySite.com/downloads/</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Downloads</td></tr>
  <tr><td>verifyText</td><td>//h2</td><td>Terms and Conditions</td></tr>
  <tr><td>clickAndWait</td><td>//input[@value="I agree"]</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Product Selection</td></tr>
</table>

That’s right – it’s an HTML table:

open http://mySite.com/downloads/
assertTitle Downloads
verifyText //h2 Terms and Conditions
clickAndWait //input[@value=“I agree”]
assertTitle Product Selection

A Selenium “test suite” is a web page with a table of links to web pages with test cases.


Selenium Webdriver

An alternate version of Selenium is more code-oriented. It provides APIs to a variety of languages allowing for very similar capabilities:

Select select = new Select(driver.findElement(
       By.tagName("select")));
select.deselectAll();
select.selectByVisibleText("Edam");

Selenium works by interacting with a “real” web browser (Firefox, Chrome) to simulate actions like clicking on or sending keystrokes to a web page element.

I’ve used Selenium Webdriver to implement some very nice web scraper applications.


Waiting

A tricky thing about testing web applications is the unknwon amount of time that may be required to respond to a lick or other interaction.

Valid tests often need to give very specific instructions to wait for pages to be loaded and for elements to become visible and clickable.

WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (
  new WebDriverWait(driver, 10))
  .until(ExpectedConditions.elementIsClickable(
            By.id("myDynamicElement")));

Waits up to 10 seconds for an expected element to load and become active.