System Testing

Steven J Zeil

Last modified: Apr 10, 2014

Contents:
1. Test Coverage
1.1 Coverage Measures
1.2 C/C++ - gcov
1.3 Java
2. Oracles
2.1 expect
2.2 *Unit
2.3 Testing GUI systems
2.4 Web systems
2.5 Selenium

1. Test Coverage

1.1 Coverage Measures

We have previously reviewed:

1.2 C/C++ - gcov

Monitoring Statement Coverage with gcov

arrayUtils.h

*  with test driver 

gcovDemo.cpp

, which reads data from a text stream (e.g., standard in), uses that data to construct arrays, and invokes each function on those arrays, printing the results of each.


Compiling for gcov Statement Coverage


Running Tests with gcov


Viewing Your Report


Sample Statement Coverage Report

     -:   69:template <typename T>
     -:   70:int seqSearch(const T list[], int listLength, T searchItem)
     -:   71:{
     1:   72:    int loc;
     -:   73:
     2:   74:    for (loc = 0; loc < listLength; loc++)
     2:   75:        if (list[loc] == searchItem)
     1:   76:            return loc;
     -:   77:
 #####:   78:    return -1;
     -:   79:}


Monitoring Branch Coverage with gcov

gcov can report on branches taken.


Reading gcov Branch Info


But What is a “Branch”?


Example: gcov Branch Coverage report

        -:   84:template <typename T>
        -:   85:int seqOrderedSearch(const T list[], int listLength, T searchItem)
        -:   86:{
        1:   87:    int loc = 0;
        -:   88:
        1:   89:    while (loc < listLength && list[loc] < searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
branch  3 taken 1
        -:   90:      {
    #####:   91:       ++loc;
branch  0 never executed
        -:   92:      }
        1:   93:    if (loc < listLength && list[loc] == searchItem)
branch  0 taken 0
call    1 returns 1
branch  2 taken 0
        1:   94:       return loc;
branch  0 taken 1
        -:   95:    else
    #####:   96:       return -1;
        -:   97:}

1.3 Java


Java Coverage Tools


Clover


JaCoCo


Example: JaCoCo in Ant

Working with our Code Annotation project, add a dependency on the JaCoCo library:


<ivy-module version="2.0">
  <info organisation="edu.odu.cs" module="codeAnnotation" revision="1.0"/>
  <publications>
    <artifact name="codeAnnotation" type="jar" ext="jar"/>
    <artifact name="codeAnnotation-src" type="source" ext="zip"/>
  </publications>
  <dependencies>
    <dependency org="de.jflex" name="jflex" rev="1.4.3"/>
    <dependency org="junit" name="junit" rev="4.10"/>
    <dependency org="org.jacoco" name="org.jacoco.ant" 
       rev="latest.integration"/>
  </dependencies>
</ivy-module>



Example: JaCoCo in Ant (cont.)

jacoco-build.xml.listing


Example: JaCoCo Report


EclEmma

Eclipse plugin for coverage tools (JaCoCo)

2. Oracles

A testing oracle is the process, person, and/or program that determines if test output is correct

2.1 expect

Covered previously, expect is a shell for testing interactive programs.

2.2 *Unit

Can we use *Unit-style frameworks as oracles at the system test level?

2.3 Testing GUI systems


Some Open Alternatives


Marathon

For Java GUIs

def test

   $java_recorded_version = "1.6.0_24"

   with_window("Simple Widgets") {
       select("First Name", "Jalian Systems")
       select("Password", "Secret")
       assert_p("First Name", "Text", "Jalian Systems")
    }
end


Jemmy

Also for Java GUIs

2.4 Web systems


Some Open Alternatives

2.5 Selenium


Selenium Scripting


Selenese

A typical scripting statement has the form

command parameter1 [parameter2]

Parameters can be


A Sample Selenium Script

<table>
  <tr><td>open</td><td>http://mySite.com/downloads/</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Downloads</td></tr>
  <tr><td>verifyText</td><td>//h2</td><td>Terms and Conditions</td></tr>
  <tr><td>clickAndWait</td><td>//input[@value="I agree"]</td><td></td></tr>
  <tr><td>assertTitle</td><td></td><td>Product Selection</td></tr>
</table>

That’s right – it’s an HTML table:

open http://mySite.com/downloads/
assertTitle Downloads
verifyText //h2 Terms and Conditions
clickAndWait //input[@value=“I agree”]
assertTitle Product Selection

A Selenium “test suite” is a web page with a table of links to web pages with test cases.


Selenium Webdriver

Provides APIs to a variety of languages allowing for very similar capabilities:

Select select = new Select(driver.findElement(
       By.tagName("select")));
select.deselectAll();
select.selectByVisibleText("Edam");


Waiting

WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (
  new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(
            By.id("myDynamicElement")));

Waits up to 10 seconds for an expected element to load