Earlier we introduced this model of the basic testing process.
In this lesson, we turn our attention to the oracle, the process for determining whether a test has failed.
Why Automate the Oracle?
Most of the effort in large test suites is evaluating correctness of outputs
The oracle is a decision procedure for deciding whether the output of a particular test is correct.
Humans (“eyeball oracles”) are notoriously unreliable
better to have tests check themselves.
Regression test suites can be huge.
Modern development methods emphasize rapid, repeated unit tests
Test-driven development: Develop the tests first, then write the code.
Debugging: if you can’t reproduce the error, how will you know when you’ve fixed it?
Output Capture and Drivers
At the unit and integration test level, we are testing functions and ADT member functions that most often produce data, not files, as their output. That data could be of any type.
How can we capture that output in a fashion that allows automated examination?
Traditional answer is to rely on the scaffolding to emit output in text form.
A more sophisticated answer, which we will explore later, is to design these tests to be self-checking.
diff
, cmp
and similar programs compare two text files byte by byte
Alternatives
grep
and similar utilities
Custom oracles
Some programs lend themselves to specific, customized oracles
pipe output from program/driver directly into a custom evaluation program, e.g.,
testInvertMatrix matrix1.in > matrix1.out
multiplyCheck matrix1.in < matrix1.out
or
testInvertMatrix matrix1.in | multiplyCheck matrix1.in
Most useful when oracle can be written with considerably less effort than the program under test
expect
is a shell for testing interactive programs.
TCL
(a portable shell script). Key expect Commands
spawn
: Launch an interactive program.
send
: Send a string to a spawned program, simulating input from a user.
expect
: Monitor the output from a spawned program. Expect takes a list of patterns and actions:
pattern1 {action1}
pattern2 {action2}
pattern3 {action3}
⋮
and executes the first action whose pattern is matched.
interact
: Allow person running expect to interact with spawned program. Takes a similar list of patterns and actions.
Sample Expect Script
Log in to other machine and ignore “authenticity” warnings.
#!/usr/local/bin/expect
set timeout 60
spawn ssh $argv
while {1} {
expect {
eof {break}
"The authenticity of host" {send "yes\r"}
"password:" {send "$password\r"}
"$argv" {break} # assume machine name is in prompt
}
}
interact
close $spawn_id
Expect: Testing a program
puts "in test0: $programdir/testsets\n"
catch {
spawn $programdir/testsets ➀
expect \ ➁
"RESULT: 0" {fail "testsets"} \ ➂
"missing expected element" {fail "testsets"} \
"contains unexpected element" {fail "testsets"} \
"does not match" {fail "testsets"} \
"but not destroyed" {fail "testsets"} \
{RESULT: 1} \{pass "testsets"} \ ➃
eof {fail "testsets"; puts "eofbsl nl"} \
timeout {fail "testsets"; puts "timeout\n"}
}
catch {
close
wait
}
➀ Launches the testsets
program
➁ Watches the output for one of the following conditions
➂ The “RESULT” line is the normal output. A result of 0 is a test case failure.
➃ A result of 1 is a test case success
The various other options are diagnostic/error messages that could appear if the program never reaches the RESULT output.
Structured Output
For unit/integration test, output is often a data structure.
Data must be serialized to generate text output and parsed to read the subsequent input
A lot of work
Similar issues can exist with the need to supply structured inputs
Repository Output
For system and high-level unit/integration tests, output may be updates to a database or other repository.
Graphics Output
For system and high-level unit/integration tests, output may be graphics
Similar issues can arise supplying GUI input
Addresses problem of capture-and-examine for structured data
Each test case is a function.
… all within the memory space of the running function
Suppose you were testing a SetOfInteger ADT and had to test the add function in isolation, you would need to know how the data was stored in the set and would have to write code to search that storage for a value that you had just added in your test. E.g.,
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
bool found = false;
for (int i = 0; i < aSet.numMembers && !found; ++i)
found = (aSet[i] == 23);
assert(found);
}
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
bool found = false;
for (int i = 0; i < aSet.numMembers && !found; ++i)
found = (aSet.data[i] == 23);
assert(found);
}
Good: captures the notion that 23 should have been added to the set
Good: requires no human evaluation
Bad: relies on underlying data structure
On the other hand, if you are testing the add and the contains function, you could use the second function to check the results of the first:
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
assert (aSet.contains(23));
}
Simpler
Robust: tests remain valid even if data structure changes
Legal: Does not require access to private
data
void testAdd (SetOfInteger startingSet)
{
SetOfInteger aSet = startingSet;
aSet.add (23);
assert (aSet.contains(23));
if (startingSet.contains(23))
assert (aSet.size() == startingSet.size());
else
assert (aSet.size() == startingSet.size() + 1);
}
Note that we
You can see the usefulness of “preserve and comapre” in this more thorough test.
void testAdd (SetOfInteger aSet)
{
for (int i = 0; i < 1000; ++i)
{
int x = rand() % 500;
bool alreadyContained = aSet.contains(x);
int oldSize = aSet.size();
aSet.add (23);
assert (aSet.contains(x));
if (alreadyContained)
assert (aSet.size() == oldSize);
else
assert (aSet.size() == oldSize + 1);
}
}
Our use of assert() in these examples has mixed results
Good: stays quiet as long as we are passing tests
Bad: testing always stops at the first failure
Bad: diagnostics are limited to file name and line number where the assertion failed.
JUnit is a testing framework that has seen wide adoption in the Java community and spawned numerous imitations for other programming languages. \bai Introduces a structure for a * suite (separately executable program) of * test cases (functions), * each performing multiple self-checking tests (assertions) * using a richer set of assertion operators
also provides support for setup, cleanup, report generation
readily integrated into IDEs
Getting Started: Eclipse & JUnit
Let’s suppose we are building a mailing list application. the mailing list is a collection of contacts.
package mailinglist;
/**
* A contact is a name and address.
* <p>
* For the purpose of this example, I have simplified matters
* a bit by making both of these components simple strings.
* In practice, we would expect Address, at least, to be a
* more structured type.
*
* @author zeil
*
*/
public class Contact
implements Cloneable, Comparable<Contact> {
private String theName;
private String theAddress;
/**
* Create a contact with empty name and address.
*
*/
public Contact ()
{
theName = "";
theAddress = "";
}
/**
* Create a contact
* @param nm name
* @param addr address
*/
public Contact (String nm, String addr)
{
theName = nm;
theAddress = addr;
}
/**
* Get the name of the contact
* @return the name
*/
public String getName()
{
return theName;
}
/**
* Change the name of the contact
* @param nm new name
*/
public void setName (String nm)
{
theName= nm;
}
/**
* Get the address of the contact
* @return the address
*/
public String getAddress()
{
return theAddress;
}
/**
* Change the address of the contact
* @param addr new address
*/
public void setAddress (String addr)
{
theAddress = addr;
}
/**
* True if the names and addresses are equal
*/
public boolean equals (Object right)
{
Contact r = (Contact)right;
return theName.equals(r.theName)
&& theAddress.equals(r.theAddress);
}
public int hashCode ()
{
return theName.hashCode() + 3 * theAddress.hashCode();
}
public String toString()
{
return theName + ": " + theAddress;
}
public Object clone()
{
return new Contact(theName, theAddress);
}
/**
* Compare this contact to another.
* Return value > 0 if this contact precedes the other,
* == 0 if the two are equal, and < 0 if this contact
* follows the other.
*/
public int compareTo (Contact c)
{
int nmcomp = theName.compareTo(c.theName);
if (nmcomp != 0)
return nmcomp;
else
return theAddress.compareTo(c.theAddress);
}
}
package mailinglist;
/**
* A collection of names and addresses
*/
public class MailingList implements Cloneable {
/**
* Create an empty mailing list
*
*/
public MailingList() {
first = null;
last = null;
theSize = 0;
}
/**
* Add a new contact to the list
* @param contact new contact to add
*/
public void addContact(Contact contact) {
if (first == null) {
// add to empty list
first = last = new ML_Node(contact, null);
theSize = 1;
} else if (contact.compareTo(last.contact) > 0) {
// add to end of non-empty list
last.next = new ML_Node(contact, null);
last = last.next;
++theSize;
} else if (contact.compareTo(first.contact) < 0) {
// add to front of non-empty list
first = new ML_Node(contact, first);
++theSize;
} else {
// search for place to insert
ML_Node previous = first;
ML_Node current = first.next;
assert (current != null);
while (contact.compareTo(current.contact) < 0) {
previous = current;
current = current.next;
assert (current != null);
}
previous.next = new ML_Node(contact, current);
++theSize;
}
}
/**
* Remove one matching contact
* @param c remove a contact equal to c
*/
public void removeContact(Contact c) {
ML_Node previous = null;
ML_Node current = first;
while (current != null && c.getName().compareTo(current.contact.getName()) > 0) {
previous = current;
current = current.next;
}
if (current != null && c.getName().equals(current.contact.getName()))
remove(previous, current);
}
/**
* Remove a contact with the indicated name
* @param name name of contact to remove
*/
public void removeContact(String name) {
ML_Node previous = null;
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
previous = current;
current = current.next;
}
if (current != null && name == current.contact.getName())
remove(previous, current);
}
/**
* Search for contacts
* @param name name to search for
* @return true if a contact with an equal name exists
*/
public boolean contains(String name) {
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
current = current.next;
}
return (current != null && name == current.contact.getName());
}
/**
* Search for contacts
* @param name name to search for
* @return contact with that name if found, null if not found
*/
public Contact getContact(String name) {
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
current = current.next;
}
if (current != null && name == current.contact.getName())
return current.contact;
else
return null;
}
/**
* combine two mailing lists
*
*/
public void merge(MailingList anotherList) {
// For a quick merge, we will loop around, checking the
// first item in each list, and always copying the smaller
// of the two items into result
MailingList result = new MailingList();
ML_Node thisList = first;
ML_Node otherList = anotherList.first;
while (thisList != null && otherList != null) {
int comp = thisList.contact.compareTo(otherList.contact);
if (comp <= 0) {
result.addContact(thisList.contact);
thisList = thisList.next;
/* if (comp == 0)
otherList = otherList.next;
[Deliberate bug ] */
} else {
result.addContact(otherList.contact);
otherList = otherList.next;
}
}
// Now, one of the two lists has been entirely copied.
// The other might still have stuff to copy. So we just copy
// any remaining items from the two lists. Note that one of these
// two loops will execute zero times.
while (thisList != null) {
result.addContact(thisList.contact);
thisList = thisList.next;
}
while (otherList != null) {
result.addContact(otherList.contact);
otherList = otherList.next;
}
// Now result contains the merged list. Transfer that into this list.
first = result.first;
last = result.last;
theSize = result.theSize;
}
/**
* How many contacts in list?
*/
public int size() {
return theSize;
}
/**
* Return true if mailing lists contain equal contacts
*/
public boolean equals(Object anotherList) {
MailingList right = (MailingList) anotherList;
if (theSize != right.theSize) // (easy test first!)
return false;
else {
ML_Node thisList = first;
ML_Node otherList = right.first;
while (thisList != null) {
if (!thisList.contact.equals(otherList.contact))
return false;
thisList = thisList.next;
otherList = otherList.next;
}
return true;
}
}
public int hashCode() {
int hash = 0;
ML_Node current = first;
while (current != null) {
hash = 3 * hash + current.contact.hashCode();
current = current.next;
}
return hash;
}
public String toString() {
StringBuffer buf = new StringBuffer("{");
ML_Node current = first;
while (current != null) {
buf.append(current.contact.toString());
current = current.next;
if (current != null)
buf.append("\n");
}
buf.append("}");
return buf.toString();
}
/**
* Deep copy of contacts
*/
public Object clone() {
MailingList result = new MailingList();
ML_Node current = first;
while (current != null) {
result.addContact((Contact) current.contact.clone());
current = current.next;
}
return result;
}
private class ML_Node {
public Contact contact;
public ML_Node next;
public ML_Node(Contact c, ML_Node nxt) {
contact = c;
next = nxt;
}
}
private int theSize;
private ML_Node first;
private ML_Node last;
// helper functions
private void remove(ML_Node previous, ML_Node current) {
if (previous == null) {
// remove front of list
first = current.next;
if (last == current)
last = null;
} else if (current == last) {
// remove end of list
last = previous;
last.next = null;
} else {
// remove interior node
previous.next = current.next;
}
--theSize;
}
}
Set up an Eclipse project with these two files
In the Eclipse Build Path, add a library: JUnit,
Right-click on the project and select New ... JUnit Test
Case
.
Give it a name (e.g., TestMailingList)
mailinglist
package as the two classesFor “Class under test”, use the Browse button and select our MailingList class
Click Next, then select the MailingList() and addContact( … ) functions
You’ll get something like this:
package mailinglist;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
public class TestMailingList {
@Test
public void testMailingList() {
fail("Not yet implemented");
}
@Test
public void testAddContact() {
fail("Not yet implemented");
}
}
Save this, and run it (right-click on the new file in the Package Explorer, and select Run As ... JUnit Test
)
Let’s implement a test for the mailing list constructor.
/**
* Create an empty mailing list
*
*/
public MailingList() {
Glancing through the MailingList interface, what would we expect of an empty mailing list?
the size()
would be zero
any contact we might search for would not be contained in the list
Change the testMailingList function to
@Test
public void testMailingList() {
MailingList ml = new MailingList(); ➀
assertFalse (ml.contains("Jones")); ➁
assertNull (ml.getContact("Smith"));
assertEquals (0, ml.size()); ➂
}
➀ Before we can test any function, we have to invoke it
➁ Notice the tests using different variations on the idea of an assertion
assert( ... )
itself remains a language primitive - we don’t use that but use these JUnit variations instead
assert( ... )
would do➂ The order of parameters in assertEquals is significant
MailingList ml = null; // new MailingList();
and run again.
Notice that a “test error” is marked differently than a “failed test”
Now let’s add a test for addContact
:
After adding a contact
the size()
should increase
the contact name should be contained in the list
we should be able to find the contact that we just added.
Testing addContact - first pass
Try this test (run it!)
@Test
public void testAddContact() {
MailingList ml = new MailingList();
Contact jones = new Contact("Jones",
"21 Penn. Ave.");
ml.addContact (jones);
assertTrue (ml.contains("Jones"));
assertFalse (ml.contains("Smith"));
assertEquals ("Jones", ml.getContact("Jones").getName());
assertNull (ml.getContact("Smith"));
assertEquals (1, ml.size());
}
It works. But it feels awfully limited.
Before we try making that test more rigorous, let’s look at how we can organize the set up of auxiliary data like our contacts:
Contact jones;
@Before
public void setUp() throws Exception {
jones = new Contact("Jones", "21 Penn. Ave.");
}
⋮
@Test
public void testAddContact() {
MailingList ml = new MailingList();
ml.addContact (jones);
⋮
The class that contains the data shared among the tests in a suite is called a fixture.
A function marked as @Before
will be run before each of the test case functions.
Why every time?
Can do cleanup in a similar fashion with “@After”
Contact jones;
Contact baker;
Contact holmes;
Contact wolfe;
MailingList mlist;
@Before
public void setUp() throws Exception {
jones = new Contact("Jones", "21 Penn. Ave.");
baker = new Contact("Baker", "Drury Ln.");
holmes = new Contact("Holmes",
"221B Baker St.");
wolfe = new Contact("Wolfe",
"454 W. 35th St.");
mlist = MailingList();
mlist.addContact(baker);
mlist.addContact (holmed);
mlist.addContact (baker);
}
Create a series of contacts
@Test
public void testAddContact() {
assertTrue (mlist.contains("Jones"));
assertEquals ("Jones",
mlist.getContact("Jones").getName());
assertEquals (4, ml.size());
}
Still seems a bit limited - we’ll address that later.
Google Test, a.k.a. gtest, provides a similar *Unit environment for C++
Download & follow instructions to prepare library
For Eclipse support, add the Eclipse CDT C/C++ Tests Runner plugin
Example
This time, the C++ version of the mailing list.
Examining the ADT
#ifndef MAILINGLIST_H
#define MAILINGLIST_H
#include <iostream>
#include <string>
#include "contact.h"
/**
A collection of names and addresses
*/
class MailingList
{
public:
MailingList();
MailingList(const MailingList&);
~MailingList();
const MailingList& operator= (const MailingList&);
// Add a new contact to the list
void addContact (const Contact& contact);
// Does the list contain this person?
bool contains (const Name&) const;
// Find the contact
const Contact& getContact (const Name& nm) const;
//pre: contains(nm)
// Remove one matching contact
void removeContact (const Contact&);
void removeContact (const Name&);
// combine two mailing lists
void merge (const MailingList& otherList);
// How many contacts in list?
int size() const;
bool operator== (const MailingList& right) const;
bool operator< (const MailingList& right) const;
private:
struct ML_Node {
Contact contact;
ML_Node* next;
ML_Node (const Contact& c, ML_Node* nxt)
: contact(c), next(nxt)
{}
};
ML_Node* first;
ML_Node* last;
int theSize;
// helper functions
void clear();
void remove (ML_Node* previous, ML_Node* current);
friend std::ostream& operator<< (std::ostream& out, const MailingList& addr);
};
// print list, sorted by Contact
std::ostream& operator<< (std::ostream& out, const MailingList& list);
#endif
Should look familiar after the Java version
Examining the Tests
#include "mailinglist.h"
#include <string>
#include <vector>
#include "gtest/gtest.h"
namespace {
using namespace std;
// The fixture for testing class MailingList.
class MailingListTests : public ::testing::Test { ➀
public:
Contact jones; ➃
MailingList mlist;
virtual void SetUp() {
jones = Contact("Jones", "21 Penn. Ave."); ➄
mlist = MailingList();
mlist.addContact (Contact ("Baker", "Drury Ln."));
mlist.addContact (Contact ("Holmes", "221B Baker St."));
mlist.addContact (Contact ("Wolfe", "454 W. 35th St."));
}
virtual void TearDown() {
}
};
TEST_F (MailingListTests, constructor) {
MailingList ml;
EXPECT_EQ (0, ml.size()); ➁
EXPECT_FALSE (ml.contains("Jones"));
}
TEST_F (MailingListTests, addContact) {
mlist.addContact (jones); ➅
EXPECT_TRUE (mlist.contains("Jones")); ➂
EXPECT_EQ ("Jones", mlist.getContact("Jones").getName());
EXPECT_EQ (4, ml.size());
}
} // namespace
Roughly similar to the JUnit tests
➀ The class provides a fixture where data can be shared among test cases
➁ Test cases are introduced with TEST_F
TEST
if you don’t provide a fixture class➂ The test cases feature assertions similar to those seen in JUnit
EXPECT
assertions allow testing to continue after failureASSERT
variations also exist that shut down testing on failure➃ Public members of the fixture class will be visible to all test cases
Boost UTF
Boost is a well-respected collection of libraries for C++.
Many of the new library components of C++11 were distributed in Boost for “beta test”.
Other, more specialized libraries will remain separately distributed.
Basic principles are similar to Google Test
Boost Test
The UTF can be added as a single header #include
or, for greater efficiency when dealing with multiple suites, compiled to form a static or dynamic library.
Easiest to start with the single header approach. Download and unpack the Boost library.
include
directory to your C++ project’s search path for system headers (#include < ... >
)
-I
compiler option, e.g., -I /home/zeil/src/boost/include
Project ... Properties ... C/C++ Build ... Settings ... GCC C++ compiler ... Includes
.For Eclipse support, use the same Eclipse CDT C/C++ Tests Runner plugin
Example
Again, the C++ version of the mailing list.
Examining the Tests
#define BOOST_TEST_MODULE MailingList test
#include "mailinglist.h"
#include <string>
#include <vector>
#include <boost/test/included/unit_test.hpp>
using namespace std;
// The fixture for testing mailing lists.
class MailingListTests { ➀
public:
Contact jones; ➁
MailingList mlist;
MailingListTests() { ➃
jones = Contact("Jones", "21 Penn. Ave.");
mlist.addContact (Contact ("Baker", "Drury Ln."));
mlist.addContact (Contact ("Holmes", "221B Baker St."));
mlist.addContact (Contact ("Wolfe", "454 W. 35th St."));
}
~MailingListTests() {
}
};
BOOST_AUTO_TEST_CASE ( constructor ) { ➄
MailingList ml;
BOOST_CHECK_EQUAL (0, mlist.size()); ➅
BOOST_CHECK (!mlist.contains("Jones"));
}
BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) { ➆
mlist.addContact (jones); ➂
BOOST_CHECK (mlist.contains("Jones"));
BOOST_CHECK_EQUAL ("Jones", mlist.getContact("Jones").getName());
BOOST_CHECK_EQUAL (4, mlist.size());
}
➀ This names the suite.
➁ The class provides a fixture where data can be shared ➂ among test cases
➄ Test cases that don’t need anything from a fixture are introduced with BOOST\_AUTO\_TEST\_CASE
➅ The assertions have different names but are similar in function and variety to JUnit and GTest
➆ Test cases that need a fixture are introduced with BOOST_FIXTURE_TEST_CASE