Automating the Testing Oracle
Steven J Zeil
Earlier we introduced this model of the basic testing process.
In this lesson, we turn our attention to the oracle, the process for determining whether a test has failed.
We will argue that the economics of testing provide a powerful incentive to automate the oracle. Traditionally, this has been done by capture-and-examine, capturing outputs and then using a separate automated process to examine and pass judgement upon those outputs.
Current practice, however, emphasizes self-checking tests, drivers that both feed input to the code under test and immediately evaluating its responses. Self-checking tests are supported by a *Unit framework. We will look at popular frameworks for both Java and C++.
1 Automating the Oracle
Why Automate the Oracle?
-
Most of the effort in large test suites is evaluating correctness of outputs
-
The oracle is a decision procedure for deciding whether the output of a particular test is correct.
-
Humans (“eyeball oracles”) are notoriously unreliable
-
better to have tests check themselves.
-
-
Regression test suites can be huge.
Thousands, tens of thousands, even hundreds of thousands of tests are not unheard of. If your idea of testing is running some code and visually inspecting the output, you can see that won’t work on this kind of scale. Regression tests have almost always been designed to check themselves. Often this was done by recording, during unit, integration, or systems test, the outputs produced by each test input. During regression testing, the same inputs are rerun, and the outputs compared to the earlier ones that had been recorded. If the outputs change, an alert is printed. If the outputs stay the same, testing moves quietly on to the next test.
-
Modern development methods emphasize rapid, repeated unit tests
It might not be obvious that self-checking is quite valuable for unit and integration testing as well. But if test outputs have to be inspected by a human, then anything more than a very few tests becomes a tedious thing to do. This motivates programmers to write very few tests and to run them infrequently, both of which are very bad ideas.
- Test-driven development: Develop the tests first, then write the code.
In fact, many of the latest trends in software development place a lot more emphasis on almost continual unit testing. In so-called agile or extreme programming, programmers are expected to write tests before implementing their functions or ADTs, and to continually rerun the tests as units are added or changed. (In effect, we get a kind of “rolling” integration test.)
In continuous integration processes, every time a programmer stops work on a unit, not only are its unit tests run, but the changes are automatically integrated into the entire program, and integration and systems tests are re-run.
But that's just not going to happen if testing is hard to do.
- Test-driven development: Develop the tests first, then write the code.
-
Debugging: if you can’t reproduce the error, how will you know when you’ve fixed it?
If a bug is reported or a new feature requested, the first step is to add new tests that fail because of the bug or missing feature. Then we will know we have successfully fixed the problem when we are able to pass that test.
The best way to be sure programmers rerun the tests on a regular basis is to make the test run part of the regular build process (e.g., build the test runs into the project make
file) and to make them self-checking.
1.1 How can oracles be automated?
Output Capture
If we are doing systems/regression tests, the first step towards automation is to capture the output:
If a program produces output files, one can self-check by creating a file representing the expected correct output, then running the program to get the actual output file and using a simple comparison utility like the Unix diff
or cmp
commands to see if the actual output is identical to the expected output.
For system-level regression tests, this is even simpler. Once we have a program that passes our system tests, we run it on those tests and save the outputs. Those become the expected output files for later regression testing. (Remember, the point of regression testing is to determine if any behavior has changed due to recent updates to the code.)
If the program updates a database, it may be possible to capture entire databases in a similar fashion. Alternatively, we write database queries to check for changes in the records most likely to have been affected by a test.
On the other hand, if the program’s main function is to present information on a screen, self-checking is very difficult. Screen captures are often not much use, because we are unlikely to want to deal with changes where, say one window is a pixel wider or a pixel to the left of where it had been in a prior test. Self-checking tests for programs like this either require extremely fine control over all possible interactive inputs and graphics device characteristics, or they require a careful “design for testability” to record, in a testing log file, information about what is being rendered on the screen. (We’ll revisit this idea later in the semester when we discuss the MVC pattern for designing user interfaces.)
Output Capture and Drivers
At the unit and integration test level, we are testing functions and ADT member functions that most often produce data, not files, as their output. That data could be of any type.
How can we capture that output in a fashion that allows automated examination?
-
Traditional answer is to rely on the scaffolding to emit output in text form.
-
A more sophisticated answer, which we will explore later, is to design these tests to be self-checking.
1.2 Examining Output
1.2.1 File Tools
diff
,cmp
and similar programs compare two text files byte by byte- used to compare expected and actual output
- useful in back-to-back testing of
- old system to its new replacement
- system before and after a bug repair
- but also used with manually generated expected output
- parameters allow special treatments of blanks, empty lines, etc.
- some versions can be used with binary files
Alternatives
- More sophisticated tests can be performed via
grep
and similar utilities- search file for data matching a regular expression
Custom oracles
-
Some programs lend themselves to specific, customized oracles
- For example, a program to invert a matrix can be checked by multiplying its input and output together — should yield the identity matrix.
-
pipe output from program/driver directly into a custom evaluation program, e.g.,
testInvertMatrix matrix1.in > matrix1.out multiplyCheck matrix1.in < matrix1.out
or
testInvertMatrix matrix1.in | multiplyCheck matrix1.in
-
Most useful when oracle can be written with considerably less effort than the program under test
1.2.2 expect
expect
is a shell for testing interactive programs.
- an extension of
TCL
(a portable shell script).
Key expect Commands
-
spawn
: Launch an interactive program. -
send
: Send a string to a spawned program, simulating input from a user. -
expect
: Monitor the output from a spawned program. Expect takes a list of patterns and actions:pattern1 {action1} pattern2 {action2} pattern3 {action3} ⋮
and executes the first action whose pattern is matched.
- Patterns can be regular expressions or simpler “glob” patterns
-
interact
: Allow person running expect to interact with spawned program. Takes a similar list of patterns and actions.
Sample Expect Script
Log in to other machine and ignore “authenticity” warnings.
#!/usr/local/bin/expect
set timeout 60
spawn ssh $argv
while {1} {
expect {
eof {break}
"The authenticity of host" {send "yes\r"}
"password:" {send "$password\r"}
"$argv" {break} # assume machine name is in prompt
}
}
interact
close $spawn_id
Expect: Testing a program
puts "in test0: $programdir/testsets\n"
catch {
spawn $programdir/testsets ➀
expect \ ➁
"RESULT: 0" {fail "testsets"} \ ➂
"missing expected element" {fail "testsets"} \
"contains unexpected element" {fail "testsets"} \
"does not match" {fail "testsets"} \
"but not destroyed" {fail "testsets"} \
{RESULT: 1} \{pass "testsets"} \ ➃
eof {fail "testsets"; puts "eofbsl nl"} \
timeout {fail "testsets"; puts "timeout\n"}
}
catch {
close
wait
}
-
➀ Launches the
testsets
program -
➁ Watches the output for one of the following conditions
-
➂ The “RESULT” line is the normal output. A result of 0 is a test case failure.
-
➃ A result of 1 is a test case success
-
-
The various other options are diagnostic/error messages that could appear if the program never reaches the RESULT output.
1.3 Limitations of Capture-and-Examine Tests
Structured Output
For unit/integration test, output is often a data structure.
-
Data must be serialized to generate text output and parsed to read the subsequent input
-
A lot of work
- Easy to omit details
- Can introduce bugs of its own
-
Similar issues can exist with the need to supply structured inputs
Repository Output
For system and high-level unit/integration tests, output may be updates to a database or other repository.
- Must be indirectly “captured” via subsequent query/access
- significant setup and cleanup effort per test
- need separate test stores
Graphics Output
-
For system and high-level unit/integration tests, output may be graphics
- very hard to capture
-
Similar issues can arise supplying GUI input
- Supplying a repeatable sequence of input events (key presses, mouse movement & clicks, etc)
- Sometimes timing-critical
2 Self-Checking Unit & Integration Tests
-
Addresses problem of capture-and-examine for structured data
-
Each test case is a function.
- That function constructs required inputs …
- and passes those inputs to the module under test …
- and examines the output …
-
… all within the memory space of the running function
In testing an ADT, we are not testing an individual function, but a collection of related functions. In some ways that makes thing easier, because we can use many of these functions to help test one another.
2.1 First Cut at a Self-Checking Test
Suppose you were testing a SetOfInteger ADT and had to test the add function in isolation, you would need to know how the data was stored in the set and would have to write code to search that storage for a value that you had just added in your test. E.g.,
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
bool found = false;
for (int i = 0; i < aSet.numMembers && !found; ++i)
found = (aSet[i] == 23);
assert(found);
}
2.1.1 What’s Good and Bad About This?
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
bool found = false;
for (int i = 0; i < aSet.numMembers && !found; ++i)
found = (aSet.data[i] == 23);
assert(found);
}
-
Good: captures the notion that 23 should have been added to the set
-
Good: requires no human evaluation
-
Bad: relies on underlying data structure
- Requires the tester to think at multiple levels of abstraction
- Test is fragile: if implementation of SetOfInteger changes, test can become useless
- Might not even compile - those data members are probably private
2.2 Better Idea: Test the Public Functions Against Each Other
On the other hand, if you are testing the add and the contains function, you could use the second function to check the results of the first:
void testAdd (SetOfInteger aSet)
{
aSet.add (23);
assert (aSet.contains(23));
}
-
Simpler
-
Robust: tests remain valid even if data structure changes
-
Legal: Does not require access to
private
data
Not only is this code simpler than writing your own search function as part of the test driver, it continues to work even if the data structure used to implement the ADT should be changed. What’s more, it is, in a sense, a more thorough test, since it tests two functions at once. Finally, there’s the simple fact that the test with the explicit loop probably won’t even compile, since it refers directly to data members that are almost certainly private.
In a sense, we have made a transition from white-box towards black-box testing. The new test case deliberately ignores the underlying structure.
2.2.1 Idiom: Preserve and Compare
void testAdd (SetOfInteger startingSet)
{
SetOfInteger aSet = startingSet;
aSet.add (23);
assert (aSet.contains(23));
if (startingSet.contains(23))
assert (aSet.size() == startingSet.size());
else
assert (aSet.size() == startingSet.size() + 1);
}
Note that we
- save the original ADT value, then
- modify a copy, and then
- compare the original and copy to see the changes
2.2.2 More Thorough Tests
You can see the usefulness of “preserve and comapre” in this more thorough test.
void testAdd (SetOfInteger aSet)
{
for (int i = 0; i < 1000; ++i)
{
int x = rand() % 500;
bool alreadyContained = aSet.contains(x);
int oldSize = aSet.size();
aSet.add (23);
assert (aSet.contains(x));
if (alreadyContained)
assert (aSet.size() == oldSize);
else
assert (aSet.size() == oldSize + 1);
}
}
2.3 assert() might not be quite what we want
Our use of assert() in these examples has mixed results
-
Good: stays quiet as long as we are passing tests
- failures easily detected by humans
-
Bad: testing always stops at the first failure
- In a large suite of many such test cases, we may be missing out on info that would be useful for debugging
-
Bad: diagnostics are limited to file name and line number where the assertion failed.
3 JUnit Testing
JUnit is a testing framework that has seen wide adoption in the Java community and spawned numerous imitations for other programming languages.
-
Introduces a structure for a
- suite (separately executable program) of
- test cases (functions),
- each performing multiple self-checking tests (assertions)
- using a richer set of assertion operators
-
also provides support for setup, cleanup, report generation
-
readily integrated into IDEs
3.1 JUnit Basics
Getting Started: Eclipse & JUnit
Let’s suppose we are building a mailing list application. the mailing list is a collection of contacts.
package mailinglist;
/**
* A contact is a name and address.
* <p>
* For the purpose of this example, I have simplified matters
* a bit by making both of these components simple strings.
* In practice, we would expect Address, at least, to be a
* more structured type.
*
* @author zeil
*
*/
public class Contact
implements Cloneable, Comparable<Contact> {
private String theName;
private String theAddress;
/**
* Create a contact with empty name and address.
*
*/
public Contact ()
{
theName = "";
theAddress = "";
}
/**
* Create a contact
* @param nm name
* @param addr address
*/
public Contact (String nm, String addr)
{
theName = nm;
theAddress = addr;
}
/**
* Get the name of the contact
* @return the name
*/
public String getName()
{
return theName;
}
/**
* Change the name of the contact
* @param nm new name
*/
public void setName (String nm)
{
theName= nm;
}
/**
* Get the address of the contact
* @return the address
*/
public String getAddress()
{
return theAddress;
}
/**
* Change the address of the contact
* @param addr new address
*/
public void setAddress (String addr)
{
theAddress = addr;
}
/**
* True if the names and addresses are equal
*/
public boolean equals (Object right)
{
Contact r = (Contact)right;
return theName.equals(r.theName)
&& theAddress.equals(r.theAddress);
}
public int hashCode ()
{
return theName.hashCode() + 3 * theAddress.hashCode();
}
public String toString()
{
return theName + ": " + theAddress;
}
public Object clone()
{
return new Contact(theName, theAddress);
}
/**
* Compare this contact to another.
* Return value > 0 if this contact precedes the other,
* == 0 if the two are equal, and < 0 if this contact
* follows the other.
*/
public int compareTo (Contact c)
{
int nmcomp = theName.compareTo(c.theName);
if (nmcomp != 0)
return nmcomp;
else
return theAddress.compareTo(c.theAddress);
}
}
package mailinglist;
/**
* A collection of names and addresses
*/
public class MailingList implements Cloneable {
/**
* Create an empty mailing list
*
*/
public MailingList() {
first = null;
last = null;
theSize = 0;
}
/**
* Add a new contact to the list
* @param contact new contact to add
*/
public void addContact(Contact contact) {
if (first == null) {
// add to empty list
first = last = new ML_Node(contact, null);
theSize = 1;
} else if (contact.compareTo(last.contact) > 0) {
// add to end of non-empty list
last.next = new ML_Node(contact, null);
last = last.next;
++theSize;
} else if (contact.compareTo(first.contact) < 0) {
// add to front of non-empty list
first = new ML_Node(contact, first);
++theSize;
} else {
// search for place to insert
ML_Node previous = first;
ML_Node current = first.next;
assert (current != null);
while (contact.compareTo(current.contact) < 0) {
previous = current;
current = current.next;
assert (current != null);
}
previous.next = new ML_Node(contact, current);
++theSize;
}
}
/**
* Remove one matching contact
* @param c remove a contact equal to c
*/
public void removeContact(Contact c) {
ML_Node previous = null;
ML_Node current = first;
while (current != null && c.getName().compareTo(current.contact.getName()) > 0) {
previous = current;
current = current.next;
}
if (current != null && c.getName().equals(current.contact.getName()))
remove(previous, current);
}
/**
* Remove a contact with the indicated name
* @param name name of contact to remove
*/
public void removeContact(String name) {
ML_Node previous = null;
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
previous = current;
current = current.next;
}
if (current != null && name == current.contact.getName())
remove(previous, current);
}
/**
* Search for contacts
* @param name name to search for
* @return true if a contact with an equal name exists
*/
public boolean contains(String name) {
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
current = current.next;
}
return (current != null && name == current.contact.getName());
}
/**
* Search for contacts
* @param name name to search for
* @return contact with that name if found, null if not found
*/
public Contact getContact(String name) {
ML_Node current = first;
while (current != null && name.compareTo(current.contact.getName()) > 0) {
current = current.next;
}
if (current != null && name == current.contact.getName())
return current.contact;
else
return null;
}
/**
* combine two mailing lists
*
*/
public void merge(MailingList anotherList) {
// For a quick merge, we will loop around, checking the
// first item in each list, and always copying the smaller
// of the two items into result
MailingList result = new MailingList();
ML_Node thisList = first;
ML_Node otherList = anotherList.first;
while (thisList != null && otherList != null) {
int comp = thisList.contact.compareTo(otherList.contact);
if (comp <= 0) {
result.addContact(thisList.contact);
thisList = thisList.next;
/* if (comp == 0)
otherList = otherList.next;
[Deliberate bug ] */
} else {
result.addContact(otherList.contact);
otherList = otherList.next;
}
}
// Now, one of the two lists has been entirely copied.
// The other might still have stuff to copy. So we just copy
// any remaining items from the two lists. Note that one of these
// two loops will execute zero times.
while (thisList != null) {
result.addContact(thisList.contact);
thisList = thisList.next;
}
while (otherList != null) {
result.addContact(otherList.contact);
otherList = otherList.next;
}
// Now result contains the merged list. Transfer that into this list.
first = result.first;
last = result.last;
theSize = result.theSize;
}
/**
* How many contacts in list?
*/
public int size() {
return theSize;
}
/**
* Return true if mailing lists contain equal contacts
*/
public boolean equals(Object anotherList) {
MailingList right = (MailingList) anotherList;
if (theSize != right.theSize) // (easy test first!)
return false;
else {
ML_Node thisList = first;
ML_Node otherList = right.first;
while (thisList != null) {
if (!thisList.contact.equals(otherList.contact))
return false;
thisList = thisList.next;
otherList = otherList.next;
}
return true;
}
}
public int hashCode() {
int hash = 0;
ML_Node current = first;
while (current != null) {
hash = 3 * hash + current.contact.hashCode();
current = current.next;
}
return hash;
}
public String toString() {
StringBuffer buf = new StringBuffer("{");
ML_Node current = first;
while (current != null) {
buf.append(current.contact.toString());
current = current.next;
if (current != null)
buf.append("\n");
}
buf.append("}");
return buf.toString();
}
/**
* Deep copy of contacts
*/
public Object clone() {
MailingList result = new MailingList();
ML_Node current = first;
while (current != null) {
result.addContact((Contact) current.contact.clone());
current = current.next;
}
return result;
}
private class ML_Node {
public Contact contact;
public ML_Node next;
public ML_Node(Contact c, ML_Node nxt) {
contact = c;
next = nxt;
}
}
private int theSize;
private ML_Node first;
private ML_Node last;
// helper functions
private void remove(ML_Node previous, ML_Node current) {
if (previous == null) {
// remove front of list
first = current.next;
if (last == current)
last = null;
} else if (current == last) {
// remove end of list
last = previous;
last.next = null;
} else {
// remove interior node
previous.next = current.next;
}
--theSize;
}
}
-
Set up an Eclipse project with these two files
- Make sure that these compile successfully with no errors.
-
In the Eclipse Build Path, add a library: JUnit,
- Choose JUnit4 if given a choice of versions 3 or 4
3.2 Structure of a JUnit Test
-
A JUnit test is a class.
-
It contains one or more “test cases” or “test functions” – member functions that return
void
, take no parameters, and are marked with@Test
. -
Each test case
- Sets up any necessary test data.
- Runs the code to be tested on that data.
- Examines the results of the test via one or more assertions.
-
A test case can
- Succeed if all assertions are true.
- Fail if any assertion is false (or if the
fail(...)
function is called explicitly). - Terminate with an error if the test case itself throws an exception.
3.2.1 Assertions
assertTrue(
condition)
: condition should be true.assertFalse(
condition)
: condition should be false.assertEquals(
x , y)
: x and y should be equal. (For objects, this is tested using theequals(...)
function. For primitives likeint
, the==
operator is used.)assertEquals(
x , y , delta)
: Floating point numbers x and y should be approximately equal, within plus or minus delta.assertArrayEquals(
array1 , array2)
: array1 and array2 should have the same length and corresponding elements in these arrays should be equal.assertSame(
obj1 , obj2)
: obj1 and obj2 should hold the same address.assertNotSame(
obj1 , obj2)
: obj1 and obj2 should not hold the same address.assertNull(
obj1)
: obj1 should be a null reference.assertNotNull(
obj1)
: obj1 should not be a null reference.
Each of these can have an optional first argument of a string that will be printed when the assertion fails. For example,
Integer pos = search(arr, x);
assertNotNull (pos);
assertNotNull ("Could not find x in arr", pos);
3.3 A JUnit Example
3.3.1 A First Test Suite
Right-click on the project and select New ... JUnit Test
Case
.
-
Give it a name (e.g., TestMailingList)
- in the same
mailinglist
package as the two classes
- in the same
-
For “Class under test”, use the Browse button and select our MailingList class
-
Click Next, then select the MailingList() and addContact( … ) functions
3.3.2 A First Test Suite (cont.)
You’ll get something like this:
package mailinglist;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
public class TestMailingList {
@Test
public void testMailingList() {
fail("Not yet implemented");
}
@Test
public void testAddContact() {
fail("Not yet implemented");
}
}
Save this, and run it (right-click on the new file in the Package Explorer, and select Run As ... JUnit Test
)
- You’ll see what failed tests look like.
- Try double-clicking on the failed cases and observe the effects on the source code listing and the Failure Trace
3.3.3 A First Test
Let’s implement a test for the mailing list constructor.
/**
* Create an empty mailing list
*
*/
public MailingList() {
Glancing through the MailingList interface, what would we expect of an empty mailing list?
-
the
size()
would be zero -
any contact we might search for would not be contained in the list
3.3.4 Testing the Constructor
Change the testMailingList function to
@Test
public void testMailingList() {
MailingList ml = new MailingList(); ➀
assertFalse (ml.contains("Jones")); ➁
assertNull (ml.getContact("Smith"));
assertEquals (0, ml.size()); ➂
}
-
➀ Before we can test any function, we have to invoke it
-
➁ Notice the tests using different variations on the idea of an assertion
assert( ... )
itself remains a language primitive - we don’t use that but use these JUnit variations instead- Use assertTrue instead of assert
- The JUnit versions don’t shut down all testing the way the primitive
assert( ... )
would do
- The first failed JUnit assertion shuts down the test case (function)
-
➂ The order of parameters in assertEquals is significant
- expected value comes first
3.3.5 Running the Constructor Test
- Run the test suite as before
- It should succeed (testAddContact will still fail)
- Change the “0” in the final test to “1”. Run again, and observe the message in the Failure Trace.
- Restore the “0”. Change the first line to
MailingList ml = null; // new MailingList();
and run again.
-
Notice that a “test error” is marked differently than a “failed test”
- Restore the first line
3.3.6 Testing addContact
Now let’s add a test for addContact
:
After adding a contact
-
the
size()
should increase -
the contact name should be contained in the list
-
we should be able to find the contact that we just added.
Testing addContact - first pass
Try this test (run it!)
@Test
public void testAddContact() {
MailingList ml = new MailingList();
Contact jones = new Contact("Jones",
"21 Penn. Ave.");
ml.addContact (jones);
assertTrue (ml.contains("Jones"));
assertFalse (ml.contains("Smith"));
assertEquals ("Jones", ml.getContact("Jones").getName());
assertNull (ml.getContact("Smith"));
assertEquals (1, ml.size());
}
It works. But it feels awfully limited.
3.4 Setting Up Tests
Before we try making that test more rigorous, let’s look at how we can organize the set up of auxiliary data like our contacts:
Contact jones;
@Before
public void setUp() throws Exception {
jones = new Contact("Jones", "21 Penn. Ave.");
}
⋮
@Test
public void testAddContact() {
MailingList ml = new MailingList();
ml.addContact (jones);
⋮
3.4.1 Fixtures
-
The class that contains the data shared among the tests in a suite is called a fixture.
-
A function marked as
@Before
will be run before each of the test case functions.- Used to (re)initialize data used in multiple tests
-
Why every time?
- Helps keep test cases independent
- During debugging, it’s common to select and run single test cases in isolation
-
Can do cleanup in a similar fashion with “@After”
3.4.2 Testing addContact - setup
Contact jones;
Contact baker;
Contact holmes;
Contact wolfe;
MailingList mlist;
@Before
public void setUp() throws Exception {
jones = new Contact("Jones", "21 Penn. Ave.");
baker = new Contact("Baker", "Drury Ln.");
holmes = new Contact("Holmes",
"221B Baker St.");
wolfe = new Contact("Wolfe",
"454 W. 35th St.");
mlist = MailingList();
mlist.addContact(baker);
mlist.addContact (holmed);
mlist.addContact (baker);
}
Create a series of contacts
3.4.3 Testing addContact - improved
@Test
public void testAddContact() {
assertTrue (mlist.contains("Jones"));
assertEquals ("Jones",
mlist.getContact("Jones").getName());
assertEquals (4, ml.size());
}
Still seems a bit limited - we’ll address that later.
3.5 Writing Expressive Tests
One goal is writing tests is to make them both easy to read and to set them up so that the messages they issue upon failure are as helpful as possible.
3.5.1 Choose the most limited assertion.
These two assertions mean the same thing:
assertTrue(string1.equals(string2));
assertEquals(string1, string2);
I would argue that the second is easier to read. Moreover, if these assertions fail, the second gives a more informative message:
import static org.junit.Assert.*;
import org.junit.Test;
public class testMessages {
String string1 = "abcdef";
String string2 = "abcZef";
@Test
public void test1() {
assertTrue (string1.equals(string2));
}
@Test
public void test2() {
assertEquals (string1, string2);
}
}
results in the messages:
test1: java.lang.AssertionError
at testMessages.java:14
test2: org.junit.ComparisonFailure: expected:<abc[d]ef> but was:<abc[Z]ef>
at testMessages.java:19
The second message is clearly more helpful.
You can alleviate this a bit by adding a message
assertTrue ("string1 is not equal to string2",
string1.equals(string2));
but, really, it’s still not as good.
Use
assertTrue
andassertFalse
only when more specific assertions do not apply. Then consider adding messages to them explaining what they are actually checking for.
For example, if what we really wanted to assert was that one string contained the other, then we would be stuck with
assertTrue ("string1 does not contain " + string2",
string1.contains(string2));
3.6 Matchers
Because the set of basic assertions is, of necessity, limited, a newer style of assertion has arisen that
- Tries to be more expressive for many common tests.
- Can be extended to new “kinds” of assertions fitted to custom ADTs.
The assertion style looks like
assertThat (object, matcher);
(Again, an optional message can be added to the front of the parameter list, but it is generally hopes that the new style reduces the need for this.)
In this new style,
assertThat (obj1, equalTo(obj2));
means the same thing as
assertEquals (obj1, obj2);
but uses the equalTo
matcher instead.
3.6.1 Core matchers
The Hamcrest Matchers include a variety of primitives for checking one value:
-
x
.equalTo(
y)
: x should be equal to y.-
Sometimes abbreviated as x
.is(
y)
.
-
-
x
.instanceOf(
class)
: x should be an instance of a particular class- Sometimes abbreviated as x
.isA(
class)
.
- Sometimes abbreviated as x
-
x
.nullValue()
: x should be a null reference. - x
.notNullValue()
: x should not be a null reference. -
x
.sameInstance(
y)
: x and y should hold the same address.
3.6.2 Core matchers – modifiers
Other core matchers are used to modify the way that matchers are applied
-
not(
matcher)
: The matcher should return false. -
allOf(
matcher1, matcher2, …)
; all of the matchers should return true. -
anyOf(
matcher1, matcher2, …)
; at least one of the matchers should return true. -
anything()
: this matcher always returns true.
With these, we can write assertions like:
assertThat(not(x.equalTo(y));
assertThat(anyOf(nullValue(x), not(x.equalTo(y)));
3.6.3 Additional Matchers
Also in the Hamcrest library, matchers for strings, numbers, iterable containers, etc.:
assertThat(x, lessThan(y));
assertThat(string1, isEmptyOrNullString());
assertThat(string1, startsWith("Hello"));
assertThat(myList, hasItem("Smith");
assertThat(myList, contains(smallerList);
assertThat(myList, containsInAnyOrder("Jones", "Doe", "Smith");
See the Matchers
class API for a full list.
4 C++ Unit Testing
4.1 Google Test
Google Test, a.k.a. gtest, provides a similar *Unit environment for C++
-
Download & follow instructions to prepare library
- gtest will be added to your project as source code
- easiest is to copy the files from fused-src/gtest/ files into a subdirectory gtest within your project.
-
For Eclipse support, add the Eclipse CDT C/C++ Tests Runner plugin
Example
This time, the C++ version of the mailing list.
- Source code is here.
- Unpack into a convenient directory.
- With Eclipse create a C++ project in that directory.
- Compile
- Run the resulting executable as a local C++ application
- Run as a *unit test:
- Right-click on the binary executable, select “Run as … Run configurations”.
- Select C/C++ Unit, click “New” button
- On the “C/C++ Testing” tab, select Tests Runner: Google Tests Runner
- Click “Run”
Examining the ADT
#ifndef MAILINGLIST_H
#define MAILINGLIST_H
#include <iostream>
#include <string>
#include "contact.h"
#include "mlIterator.h"
/**
A collection of names and addresses
*/
class MailingList
{
public:
typedef MLConstIterator iterator;
typedef MLConstIterator const_iterator;
MailingList();
MailingList(const MailingList&);
~MailingList();
const MailingList& operator= (const MailingList&);
// Add a new contact to the list
void addContact (const Contact& contact);
// Does the list contain this person?
bool contains (const Name&) const;
// Find the contact
const Contact& getContact (const Name& nm) const;
//pre: contains(nm)
// Remove one matching contact
void removeContact (const Contact&);
void removeContact (const Name&);
// combine two mailing lists
void merge (const MailingList& otherList);
// How many contacts in list?
int size() const;
bool operator== (const MailingList& right) const;
bool operator< (const MailingList& right) const;
// Provides iterator over contacts
const_iterator begin() const;
const_iterator end() const;
private:
struct ML_Node {
Contact contact;
ML_Node* next;
ML_Node (const Contact& c, ML_Node* nxt)
: contact(c), next(nxt)
{}
};
ML_Node* first;
ML_Node* last;
int theSize;
// helper functions
void clear();
void remove (ML_Node* previous, ML_Node* current);
friend std::ostream& operator<< (std::ostream& out, const MailingList& addr);
};
// print list, sorted by Contact
std::ostream& operator<< (std::ostream& out, const MailingList& list);
#endif
Should look familiar after the Java version
Examining the Tests
#include "mailinglist.h"
#include <string>
#include <vector>
#include "gtest/gtest.h"
namespace {
using namespace std;
// The fixture for testing class MailingList.
class MailingListTests : public ::testing::Test { ➀
public:
Contact jones; ➃
MailingList mlist;
virtual void SetUp() {
jones = Contact("Jones", "21 Penn. Ave."); ➄
mlist = MailingList();
mlist.addContact (Contact ("Baker", "Drury Ln."));
mlist.addContact (Contact ("Holmes", "221B Baker St."));
mlist.addContact (Contact ("Wolfe", "454 W. 35th St."));
}
virtual void TearDown() {
}
};
TEST_F (MailingListTests, constructor) {
MailingList ml;
EXPECT_EQ (0, ml.size()); ➁
EXPECT_FALSE (ml.contains("Jones"));
}
TEST_F (MailingListTests, addContact) {
mlist.addContact (jones); ➅
EXPECT_TRUE (mlist.contains("Jones")); ➂
EXPECT_EQ ("Jones", mlist.getContact("Jones").getName());
EXPECT_EQ (4, ml.size());
}
} // namespace
Roughly similar to the JUnit tests
-
➀ The class provides a fixture where data can be shared among test cases
- This is optional, but common
-
➁ Test cases are introduced with
TEST_F
- First argument identifies the suite (same as fixture name)
- Second argument names the test case
- The combination must be unique
- Use
TEST
if you don’t provide a fixture class
-
➂ The test cases feature assertions similar to those seen in JUnit
EXPECT
assertions allow testing to continue after failureASSERT
variations also exist that shut down testing on failure
-
➃ Public members of the fixture class will be visible to all test cases
- ➄ and are assigned the values during SetUp, run before each test case
- ➅ You can see the fixture members used here
4.2 Boost Test Framework
Boost UTF
Boost is a well-respected collection of libraries for C++.
-
Many of the new library components of C++11 were distributed in Boost for “beta test”.
-
Other, more specialized libraries will remain separately distributed.
- This include another popular *Unit framework for C++, the Boost Unit Test Framework (UTF).
-
Basic principles are similar to Google Test
- Also has some support for building trees of test suites.
Boost Test
-
The UTF can be added as a single header
#include
or, for greater efficiency when dealing with multiple suites, compiled to form a static or dynamic library. -
Easiest to start with the single header approach. Download and unpack the Boost library.
- Add the Boost
include
directory to your C++ project’s search path for system headers (#include < ... >
)- If you re using a makefile, add the
-I
compiler option, e.g.,-I /home/zeil/src/boost/include
- In Eclipse, this is
Project ... Properties ... C/C++ Build ... Settings ... GCC C++ compiler ... Includes
.
- If you re using a makefile, add the
- Add the Boost
-
For Eclipse support, use the same Eclipse CDT C/C++ Tests Runner plugin
Example
Again, the C++ version of the mailing list.
- Source code is here.
- Unpack into a convenient directory.
- With Eclipse create a C++ project in that directory.
- Add path to Boost include directory
- Compile
- Run the resulting executable as a local C++ application
- Run as a *unit test:
- Right-click on the binary executable, select “Run as … Run configurations”.
- Select C/C++ Unit, click “New” button
- On the “C/C++ Testing” tab, select Tests Runner: Boost Tests Runner
- Click “Run”
Examining the Tests
#define BOOST_TEST_MODULE MailingList test
#include "mailinglist.h"
#include <string>
#include <vector>
#include <boost/test/included/unit_test.hpp>
using namespace std;
// The fixture for testing mailing lists.
class MailingListTests { ➀
public:
Contact jones; ➁
MailingList mlist;
MailingListTests() { ➃
jones = Contact("Jones", "21 Penn. Ave.");
mlist.addContact (Contact ("Baker", "Drury Ln."));
mlist.addContact (Contact ("Holmes", "221B Baker St."));
mlist.addContact (Contact ("Wolfe", "454 W. 35th St."));
}
~MailingListTests() {
}
};
BOOST_AUTO_TEST_CASE ( constructor ) { ➄
MailingList ml;
BOOST_CHECK_EQUAL (0, mlist.size()); ➅
BOOST_CHECK (!mlist.contains("Jones"));
}
BOOST_FIXTURE_TEST_CASE (addContact, MailingListTests) { ➆
mlist.addContact (jones); ➂
BOOST_CHECK (mlist.contains("Jones"));
BOOST_CHECK_EQUAL ("Jones", mlist.getContact("Jones").getName());
BOOST_CHECK_EQUAL (4, mlist.size());
}
-
➀ This names the suite.
-
➁ The class provides a fixture where data can be shared ➂ among test cases
- Optional, but common
- Simpler in Boost than in Google
- Can be any class.
- Initialization is done in the class constructor instead of in a special function. ➃
-
➄ Test cases that don’t need anything from a fixture are introduced with
BOOST\_AUTO\_TEST\_CASE
- Argument names the test case
-
➅ The assertions have different names but are similar in function and variety to JUnit and GTest
- A full list is here.
-
➆ Test cases that need a fixture are introduced with
BOOST_FIXTURE_TEST_CASE
- Second argument identifies the fixture class
- Public members of the fixture class will be visible to test cases ➂
4.3 CppUnitLite
CppUnitLite is my own C++ test framework, with features:
- Supports the newer “matcher” style of JUnit/Hamcrest
- Lightweight – requires adding one
.h
and one.cpp
file to a project. - Emulates GoogleTest when run within Eclipse
- Portable: works in Linux, MacOs, and Windows with CygWin or MingW
- Unit tests are time-limited by default (Linux/MacOs only)
- But timing functions are turned off when running in a debugger.
- When running in a debugger, a breakpoint is generated automatically on test assertion failure.
Future versions of this framework are intended to offer a mocking framework that is simpler and more intuitive than currently available elsewhere.
4.3.1 Example
#include "unittest.h" ➀
#include "mailinglist.h"
#include <string>
#include <vector>
using namespace std;
Contact jones;
MailingList mlist;
void setUp()
{
jones = Contact("Jones", "21 Penn. Ave.");
mlist = MailingList();
mlist.addContact (Contact ("Baker", "Drury Ln."));
mlist.addContact (Contact ("Holmes", "221B Baker St."));
mlist.addContact (Contact ("Wolfe", "454 W. 35th St."));
}
UnitTest (constructor) { ➁
MailingList ml;
assertThat (ml.size(), is(0)); ➂
assertFalse (ml.contains("Jones"));
}
UnitTest (addContact) {
setup(); ➃
mlist.addContact (jones);
assertTrue (mlist.contains("Jones"));
assertThat (mlist.getContact("Jones").getName(), is("Jones"));
assertThat (ml.size(), is(4));
assertThat (ml, hasItem(jones)); ➄
}
- ➀ Imports the unit test framework.
- ➁ New test cases are introduced by
UnitTest
orUnitTestTimed
.All tests are time-limited by default (not supported in Windows) but
UnitTestTimed
overrides the default. -
➂ A typical assertion, in the newer “matcher” style favored by JUnit.
- ➃ In general,
UnitTestLite
aovoids creating special constructs for things that can easily be done by normal programming, such as running a setup function at the start of a test. - ➄ An example of testing a collection.
hasItem
searches any data structure that provides iterators.This could also be written as
assertThat (jones, isIn(ml));