Eric Gufford has been a professional developer on Wall Street for 20 years, specializing in Enterprise scalable systems for both equities and fixed income securities. He is also on the ANSI C++ Technical Committee and the author of Pure C++ (SAMS, 1999). Contact Eric at [email protected].
Unit testing is one of those things that everybody preaches, but precious few actually practice. There are many explanations for this phenomenon, but it generally boils down to the following.
- It is hard. Many classes don't lend themselves easily to the unit-testing paradigm. Thus, it becomes easy to say "my code is too complicated for this," which eventually becomes a self-fulfilling prophecy.
- It is time consuming. Unit-testing frameworks can be difficult to set up, difficult to learn, and generally take way too much time to implement, thus squandering precious development time better served solving business problems.
- It is complicated. There are many different frameworks on the market, some open source, others commercial. They are all large, requiring copious resources, and are generally unwieldy. They cause code bloat in the makefile and can't be readily used in an IDE.
I used to subscribe to these types of excuses (some more than others) but I've found over the years that they're patently wrong. Many trees have been dedicated to dealing with the first excuse, but the other two have been pretty much valid in my experience—at least until recently. I was recently tasked with finding a unit-testing framework for our development group. My marching orders were simple: The product must be robust; it must be easy to incorporate into our existing code base (several million lines of code, some over 10 years old); and it must be cheap (open source, if possible).
Being both a C++ and Java developer, I immediately thought of JUnit, which is the 'gold standard' for Java unit testing. In short order, I found several open-source products that work similarly to JUnit. Each product I tried missed the mark. Some were complex to install and use (for example, CPPUnit), others were just too lightweight for the task (for example, CPPUnitLite), and some didn't handle both UNIX and Windows/MFC (for example, GUnit).
Then I found an article on the subject by Noel Llopis that covered much of the same ground (http://www.gamesfromwithin.com/ articles/0412/000061.html). In essence, Noel is a proponent of CxxTest, and with good reason. As a result of his analysis, I chose to try CxxTest for myself, with the intent of adopting this framework for my group's unit-testing needs. As a proof of concept (POC), I chose an existing project as the source to be unit tested, as this project was fairly simple and the classes were designed with unit testing in mind. To show just how easy this framework is to use, I will cover the aforementioned POC in this article.
How Does It Work?
In short, CxxTest is a framework that uses Perl to construct robust testing classes. Essentially, you include a header file (called TestSuite.h) and declare a class that inherits from CxxTest::TestSuite, the base class of all unit tests, as shown in Listing 1. In the child class, you insert all the code required for the various tests the class is meant to perform. From here, a Perl (or Python) script is run that actually writes the class that will implement the tests. Thus, the Perl script does the heavy lifting, relieving the developer from having to worry about the arcane mechanics of the framework.
Once the testing class is declared, each test method can be defined. Each method must start with a lower case 'test' in order for the Perl script to see it. For example, in the class declaration in Listing 1, a test method in my POC designed to test a parser class was called testMain(). Each test method must take no parameters nor return anything. The testing class may contain member variables, and there are mechanisms in the base class for setting up and tearing down objects that are required for use throughout the test; they will be discussed later in this article.
The general idea is that the testing class provides a way to logically group related unit tests together into a cohesive whole. Each test method stands alone, called in succession on a first come, first served basis. Thus, each method should be a well defined, well thought out unit test. An easy way to work with this is to simply list all the tests you would perform on a class, group them together into logical units, and then implement these logical units as test classes. Listing 2 contains a testing class that shows how this would work.
There are no real restrictions on the testing methods defined in a test class; they may instantiate classes, interact with the filesystem (or databases for that matter), and generally do anything that makes sense. The testMain() method in Listing 1 simply declares an instance of the class to be tested (called Main_c in this case), sets up two formal parameters naming the input and output files the class is to use, and then invokes Main_c::GetData to perform the test. This method returns an object of type Parser_c, which contains all the data that was parsed from the aforementioned input file. The test then simply dumps the contents of the parser's data into a file called Parser.out.
This test is very simple, containing no exception handling, even though the class being tested makes extensive use of exception specifications and uses a fairly robust exception class hierarchy. Obviously, when testing production quality systems, you would want to wrap exception handlers around all method calls just to ensure proper coverage of the test case, the idea being that when a tested method throws, the exception handler would record this fact and decide whether the exception was expected or not. For example, it would make perfect sense to test whether a given condition generates an expected exception. If an exception wasn't generated, the test would fail.
Logging
CxxTest includes a series of logging macros that can be used to record test status (such as fail or warning) as well as trace information, called TS_FAIL, TS_WARNING, and TS_TRACE. Passing a test is as simple as returning from the function.
These macros not only print to standard out but they provide a direct, unambiguous way to declare the state of the test. For example, TS_FAIL will fail the test and stop execution of the testing function. The TS_WARNING macro will not end the test function as TS_FAIL would; it will simply write a message to standard out and mark the test as not succeeding. The TS_TRACE macro will write to standard out, but not change the state of the test.
There are also a number of assertions that you can use for a variety of purposes. There are the standard equality operators (TS_ASSERT_EQUALS, TS_ASSERT_LESS_THAN, TS_ASSERT_LESS_THAN_EQUALS), variations for data equality (TS_ASSERT_SAME_DATA), and even a variation that supports exceptions (TS_ASSERT_THROWS_ANYTHING, TS_ASSERT_THROWS_NOTHING, TS_ASSERT_THROWS _EQUALS). Pretty much anything you would want to assert is covered in one way or another. To reverse the logic, you need only precede them with the not operator (for example, !TS_ASSERT_EQUALS) for the desired effect. There is even a variation of the foregoing macros that will print a user-defined message. These macros start with TSM (ETSM for exception-throwing variants) instead of TS (and ETS) versions.
Test Fixtures
CxxTest supports the concept of fixtures, which are objects that get constructed once at the beginning of the test and are torn down once at the end of the test. By defining these fixtures in one place, you can simplify the code in each test, while being assured that each test will use the same objects, constructed the same way. Thus, the code required to set up the test (and clean up afterwards) only needs to be tested once itself—a significant time- and labor-saving mechanism.
CxxTest supports the concept of suite-level fixtures, which are objects that get constructed once at the beginning of the test suite and are torn down once at the end of the test suite. This is quite helpful in cases where the object being tested is difficult or time consuming to construct, or where support classes are required that can be shared across tests in a suite. These static functions are called createSuite() and destroySuite(). These functions will be called once when the suite is first created, and once when the suite is being disposed of.
Output
CxxTest can produce output in many different ways. This is specified on the command line of the Perl script that generates the class. The default mechanism is designed for use within an IDE and is controlled via the --runner option. This can use regular formatting such as stdout; print line numbers like Visual Studio expects; or it can simply provide a True/False result. This can be combined with the --gui option to describe the type of user interface you would like in the testing application. Options are Win32, X11, or QT. In either case, the behavior is similar.
Once the testing program is built, the resulting executable can be run on demand. On the Windows platform, the program reports its output through a progress-bar-like mechanism that simply shows failures as red, warnings as yellow, and success as green. At the end of the run, the progress bar will show the overall result of the test. If any failures were found, the bar will show red; any warnings (but no failures) and the bar is yellow; otherwise the bar is green. This is very helpful when running long tests over a period of time.
Tying It All Together
Ideally, unit tests should be incorporated into a makefile, such that every time the program is built, the unit tests are run. CxxTest comes with makefiles for a variety of platforms and compilers, such as Linux/GNU, UNIX/GNU, and Windows/Visual Studio. Under the Windows configuration, there are sample applications that will tie in the makefiles into your project definitions, such that the entire testing application is built, and all tests run, as part of the build. If any tests fail, the entire build fails; if any test produces a warning, the build system will report the warning as well. Simply setting the failure level to fail on warnings will cause the entire build to fail on a single warning.
Thus, it is possible to incorporate these unit tests into the overall build of the system being tested. This can be done either by creating a dependant project, directly calling the makefiles, or by scripting. In any case, you can be sure that each time the project is built, the unit tests will be run.
Conclusion
In the past, unit testing was difficult, but with a good unit-testing tool you can now create a unit-test harness that can help identify bugs and ensure software quality.
When I started the POC project, I chose an application that was simple, designed for unit testing, and fairly well tested. Despite finishing in a couple of days, I found that the source code being tested had six nontrivial bugs that otherwise would not have been found—in code that I had already hand debugged in the IDE!
I am now a proponent not only of unit testing in general, but also of CxxTest in particular. I have recommended that my development group adopt this product as its official unit-testing framework.
Check out the online documentation and download the latest version of CxxTest at http://cxxtest.sourceforge.net/.
CUJ
