System Testing

Steven J Zeil

Last modified: Apr 10, 2014

1. Test Coverage

1.1 Coverage Measures

We have previously reviewed:

1.2 C/C++ - gcov

Monitoring Statement Coverage with gcov

arrayUtils.h

*  with test driver 

gcovDemo.cpp

, which reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.

Compiling for gcov Statement Coverage

Running Tests with gcov

Viewing Your Report

Sample Statement Coverage Report

     -:   69:template <typename T>
     -:   70:int seqSearch(const T list[], int listLength, T searchItem)
     -:   71:{
     1:   72:    int loc;
     -:   73:
     2:   74:    for (loc = 0; loc < listLength; loc++)
     2:   75:        if (list[loc] == searchItem)
     1:   76:            return loc;
     -:   77:
 #####:   78:    return -1;
     -:   79:}

Monitoring Branch Coverage with gcov

gcov can report on branches taken.

Reading gcov Branch Info

But What is a “Branch”?

Example: gcov Branch Coverage report

        -:   84:template <typename T>
        -:   85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
        -:   86:{
        1:   87:    int loc = 0;
        -:   88:
        1:   89:    while (loc < listLength && list[loc] < searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
branch  3 taken 1
        -:   90:      {
    #####:   91:       ++loc;
branch  0 never executed
        -:   92:      }
        1:   93:    if (loc < listLength && list[loc] == searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
        1:   94:       return loc;
branch  0 taken 1
        -:   95:    else
    #####:   96:       return -1;
        -:   97:}

1.3 Java

Java Coverage Tools

Clover

JaCoCo

Example: JaCoCo in Ant

Working with our Code Annotation project, add a dependency on the JaCoCo library:


<ivy-module version="2.0">
  <info organisation="edu.odu.cs" module="codeAnnotation" revision="1.0"/>
  <publications>
    <artifact name="codeAnnotation" type="jar" ext="jar"/>
    <artifact name="codeAnnotation-src" type="source" ext="zip"/>
  </publications>
  <dependencies>
    <dependency org="de.jflex" name="jflex" rev="1.4.3"/>
    <dependency org="junit" name="junit" rev="4.10"/>
    <dependency org="org.jacoco" name="org.jacoco.ant" 
       rev="latest.integration"/>
  </dependencies>
</ivy-module>


Example: JaCoCo in Ant (cont.)

jacoco-build.xml.listing

Example: JaCoCo Report

EclEmma

Eclipse plugin for coverage tools (JaCoCo)

2. Oracles

A testing oracle is the process, person, and/or program that determines if test output is correct

2.1 expect

Covered previously, expect is a shell for testing interactive programs.

2.2 *Unit

Can we use *Unit-style frameworks as oracles at the system test level?

2.3 Testing GUI systems

Some Open Alternatives

Marathon

For Java GUIs

def test

   $java_recorded_version = "1.6.0_24"

   with_window("Simple Widgets") {
       select("First Name", "Jalian Systems")
       select("Password", "Secret")
       assert_p("First Name", "Text", "Jalian Systems")
    }
end

Jemmy

Also for Java GUIs

2.4 Web systems

Some Open Alternatives

2.5 Selenium

Selenium Scripting

Selenese

A typical scripting statement has the form

command parameter1 [parameter2]

Parameters can be

A Sample Selenium Script

<table>
  <tr><td>open</td><td>http://mySite.com/downloads/</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Downloads</td></tr>
  <tr><td>verifyText</td><td>//h2</td><td>Terms and Conditions</td></tr>
  <tr><td>clickAndWait</td><td>//input[@value="I agree"]</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Product Selection</td></tr>
</table>

That’s right – it’s an HTML table:

open http://mySite.com/downloads/
assertTitle Downloads
verifyText //h2 Terms and Conditions
clickAndWait //input[@value=“I agree”]
assertTitle Product Selection

A Selenium “test suite” is a web page with a table of links to web pages with test cases.

Selenium Webdriver

Provides APIs to a variety of languages allowing for very similar capabilities:

Select select = new Select(driver.findElement(
       By.tagName("select")));
select.deselectAll();
select.selectByVisibleText("Edam");

Waiting

WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (
  new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(
            By.id("myDynamicElement")));

Waits up to 10 seconds for an expected element to load