Five Questions With Robert Musson

Robert Musson has been doing software for twenty-five years now. He spent fifteen of those years at Teradyne working on telecommunications products. He also helped deploy the Team Software Process (TSP) there, which experience he enjoyed so much he later joined the TSP Initiative at the Software Engineering Institute of Carnegie-Mellon University. These days Robert is a development lead on Microsoft's Windows Core Security Test Team, where he manages what he calls the "Department of Statistical Distortions". (Wonder what he means by that? See his chapters in The Practical Guide to Defect Prevention for further details. See my review of said chapters elsewhere on DDJ.)

Here is what Robert has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
RM: My first introduction to formal testing techniques was in a MSSE class at IIT. It was the usual stuff: boundary case analysis, the V model, and so on. But I don’t think it gave me a good appreciation for what testing really means and what it should mean. Testing in software is really all about quality control, that is, finding defects in the software. This is a very dangerous view since it leads to the thought that testing is where quality comes from. The mindset then becomes that testing, and quality, are reactive in nature when in fact testing needs to be proactive. We need methods to predict the quality of the products and to focus effort on assuring the product is ready rather than just pounding away until it seems to be ready.

DDJ: What is the most interesting bug you have seen?
RM: I’ve seen lots of interesting bugs that take down critical systems for extended periods. What makes them interesting to me is the simplicity of the mistake. For example, many years ago someone once put a comment in the code with the "/*" format and forgot the closing "*/". This accidentally commented out an error path. The code worked under all except the most extreme conditions, which of course were impossible to test for. The most insidious bugs are typically simply goofs on the part of the developer. Yet our testing effort tend to focus on ever more complicated techniques thinking that it is the complex race condition or logic error that must be discovered. We often forget about simple things to find simple problems. Code inspections, for example, are often thought of as a development step, but they also should be thought of as a form of test, or rather, quality assurance. In the cited example, the problem was found by someone just looking at the code and not by any complex test that reproduced the problem. Testers should be familiar with the details of development simply so they can discover defects by thinking rather than by brute force. That is really the point of modeling and simulation. Think about the issue and discover problems before they occur.

DDJ: How would you describe your testing philosophy?
RM: Testing is not about finding bugs but about proving readiness. If testers are finding bugs, that indicates a problem with developing. The job of testing is to prove the software should be shipped. Unfortunately, the mindset of finding bugs causes us to ship once our test cases have been run. As a result, software does not so much ship as escape. Most teams have no idea how many undiscovered defects exist in the software nor do they know the potential impact to the customer. As a general rule, teams do not understand the potential costs associated with fixing the undiscovered bugs. This is the software equivalent of product warranty costs. I am really big on tying the test efforts to system costs. A typical software company spends at least as much maintaining the product as they did developing it. Large product teams then begin to collapse under the weight of the rework costs. I call this a software black hole from which no useful work can escape. Only by controlling the risk associated with releasing software will organizations be able to control repair costs.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RM: We tend to focus on advanced testing techniques. Of course new test technologies are important, but I would like to see all testers trained in statistical techniques and measurements. Simple data tracking, like defects per class, provide invaluable information but only when proper analysis is done. We sometimes see people react to data because it seems bad. Only when we do proper analysis can we determine anything meaningful. There are many things about the development process that are counterintuitive and cause us to make wrong decisions. The job of test is to aid the decision making process, and in many cases, only the test team can provide the information. Yet we often ignore the data, or else react only to the last data point and forget long term trend information. I equate proper data management and analysis to good engineering. Testers should use good engineering techniques and we might be able to spread that type of discipline to other aspects of development.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RM: Automation. I say that tongue in check, but humor often has some element of truth in it. There is a current push that everything needs to be automated and more automation is better. Yet when the analysis is done and we look at costs and impacts to the system, automation can be counterproductive. We need the right automation and the right amount of it. Too much is as bad as too little and in some cases worse. This is where simulation and modeling becomes a key to managing costs. We found that we were actually increasing costs by simply automating everything, and what was worse, we actually had lower product throughput without increasing quality. That was one of those counterintuitive things I mentioned earlier. Some areas are better left to good old ad hoc testing. Don’t get me wrong, some areas require automation as the only way to assure quality. But simply writing 2 or 3 times more test code than product code does not improve the quality and will likely end up greatly increasing the costs. This is where proper planning comes in and having your best engineers as testers can really cut product costs and increase quality. There is a view that higher quality means higher costs, and that is just wrong.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink