What To Know?
Traditionally, senior executives hand off the responsibility for monitoring software quality to the company CIO. They expect him/her to report periodically on the status of various projects and initiatives and to provide some assurance of software quality. Under new models of governance, however, this delegation of responsibility is being supplanted by a structure that requires the CIO to report software quality metrics on a regular basis to the company CFO.
The choice of metrics to monitor is important, as the data must correlate to software quality and it must be comprehensible to the CFO and other executives. Typically, three measures provide useful and intelligible quality-tracking monitors:
- Code test coverage. How much code is exercised by tests? This number, generally given as a percent of total lines of code, is imperfect but nonetheless a good proxy for the level of confidence (or level of risk) associated with that software. The better the coverage, the greater the confidence that detectable bugs have been removed. The metric, you'll note, says nothing about how the test results turned out, but it does indicate how much of the code the developers are really focused on. Code coverage numbers need to be broken out by project. Old, stable code often has much lower test coverage than new code under development. So, consumers of this metric need to know both the coverage numbers and the type of project they apply to.
CIOs and CFOs should agree on acceptable variations in expected coverage. Even on greenfield projects, for example, 100 percent code coverage is nearly impossible to achieve; there is a point of diminishing returns as with any other engineering activity. So a reasonable goal should be set and monitored. Dips below the agreed-upon threshold are events that should generate specific actions (such as a briefing or a report and, depending on the reason, an estimate as to when the numbers will return to required levels). On maintenance code, the target range will certainly be lower than on newer projects. Here, too, coverage levels should be established and enforced.
- Defect rate. The number of defects -- generally expressed either in absolute terms or per thousand of lines of code (KLOC) -- is the complement to code coverage. Whereas the coverage indicates how much code was tested, the defect rate communicates the results of those tests. Frequently, development teams rely on results from functional tests and unit tests that are run as regression suites over the code base. The desirable number of defects, of course, is zero. But the acceptable range is difficult to identify and requires understanding of a project's status. For example, it is quite common to add a new module to a program. That module can easily require the rewriting of substantial portions of existing code. In such a situation, the addition of the new module will lead to a significant but temporary jump in the defect rate. This jump is expected and so is not a cause for alarm -- as long as all parties are aware of what is happening. However, the trends in defect rates are important. On new projects, the defect rate should be dropping steadily as the software moves closer to deployment. If this is not happening, the CIO owes senior management an explanation of the problem (and, likely, a revision of the timeline).
Before discussing the third measure, let's see where these two measures take us. Consider that you are senior management at a company that is about to deploy an important package that was developed in-house. In the traditional scenario, your level of confidence in the software probably depends mainly on your level of confidence in your CIO. All project data and explanation comes from the CIO, anyway, so you know your view into the project has already been conditioned by the IT reporting structure. At deployment time, you're forced to hope that the job's been done right.
Now consider the newer model in which you've been using a dashboard to track coverage and defect counts with precision. This information comes directly from automated processes, so it's the same raw data that IT sees. You have hard numbers and you have reports from the CIO about problems that have occurred along the way. You now have a basis of comparison with previous projects and a good sense of what the issues have been as the project moved ahead. You also have a feel for how much testing has occurred and the results of those tests. Aren't you in a much better position with respect to your knowledge of company systems and the likelihood of a successful project? A third metric can be used to give you even more data and increased confidence in your software.
- Risk. In software projects, risk correlates well to the product of code complexity and code usage. This is a common sense observation: Highly complex code is more likely to have bugs, and be the source of new bugs when it is modified. And the more often a code module is used, the more side effects a bug in that code will have. Hence, it is well worth tracking the risk associated with each module (usually the method, in Java). There are several industry-standard measures for quantifying complexity, of which the most widely used is for cyclomatic complexity (also called "McCabe's metric"). This metric measures the number of different execution paths through a piece of code. Unlike the other measures, cyclomatic complexity has hard and fast numbers whose value is independent of the type of project and the maturity of the code base. Table 1 shows the range of values and the associated risk.
Projects should aim to have 75 percent or more of their modules have a complexity of less than 10, and allow very few modules with a complexity above 20. Anything more than this is likely to indicate convoluted code that is difficult to verify. In other words, it is risky code. If it is in a very used routine, the risk increases commensurately.
These three measures -- code coverage, defect rate, and risk -- paint a fairly accurate view of the quality of a company's software-development process. They are, however, only the basics. Other measures such as load tests, open issues in the defect tracking system, and other criteria can fill out the picture even more.
It's important to note that tracking these measures is not a guarantee of success. In the same sense, no metric or group of metrics provides guarantees of ultimate quality. A program can enjoy 100 percent test coverage, no defects, no open issues, and simple code routines yet still fail badly. However, such failures tend to be rare. And statistically, projects with less-favorable metrics will fail more often. So, the measures do provide visibility into the overall quality of software development at the firm.
Their value increases when metrics on current projects can be compared with those of projects past. So, to establish the maximum context for the results, the data should be kept and used to validate new numbers.
