Five Questions With Douglas Hoffman

Douglas Hoffman owns and operates Software Quality Methods, where he does test planning, managing, and training for companies such as Hewlett-Packard, Palm Computing, and Sun Microsystems. You may have seen him talk at PNSQC last week, which is the latest of many papers and presentations he has given over the last many years. Douglas teaches testing at the University of California at Santa Cruz Extension, is a founding member of the Association for Software Testing, and is a Fellow of the American Society for Quality. Here is what Douglas has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DH: This is hard to answer. Besides being a long time ago (I started programming using coding sheets and 36 hour turn-around time from submission to getting the printout - I didn't even see any punch cards), my introduction to testing was slow immersion over 8 or so years as a developer. It wasn't until the early 70's that I did any work that I thought of as being testing, and then it was system level testing on a computer system manufacturing floor.

I wrote the production instructions for technicians to assemble and test customized minicomputer systems. Because my audience was production technicians, the tests had clear pass/fail criteria. I learned something back then that is still absent in some software testing - there has to be some way of determining a verdict when the test results come back. There has to be some way to tell pass from fail. A test needs to show whether or not the software behaved well. Someone also needs to care about whether or not it worked - tests aren't helpful when, if they fail, nobody is going to do anything. That just seems like a frustrating waste of time.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
DH: I'm still surprised and disappointed by how may managers and testers think testing improves quality. There seems to be a perpetual myth that just knowing about a problem fixes it. "Why didn't Testing find that defect?" isn't the first or most important question to ask. Testing didn't create the defect. Testing can't find all the defects. Testing doesn't take them out. The information provided by testing is critical for understanding the quality and to avoid too many embarrassments. But, defects are created before the program got into testing. That's where the most important questions are.

DDJ: What is the most interesting bug you have seen?
DH: I've been fascinated so many ways by so many defects. How simply and elegantly one incorrect bit in the code can so thoroughly trash the work done by the other 100 Megabytes of the program. Which is the most interesting? The comment change that shut down the 100+ developers for a week. The rushed, untested, one line fix that the VP forced into production (to shave a week off of the release) that overnight rendered 16,000 desktop systems useless (until the technicians could get around to individually reloaded them). The square root problem that caused just two of the 4 Billion computations to be wrong.

Probably at the top of my list is a data-dependent DMA error. (Direct Memory Access was a hardware port that sent or received blocks of data without tying up the CPU. It was used especially with disk and tape.) The problem was due to a combination of network speed, printed circuit board layout, power supply noise, and specific data values that led to different bits arriving at the disk drive than were in the memory. I spent a week with my head plastered against an oscilloscope hood, looking at bits fly by. I would tweak the data, look at the bits. Probe the next data line, probe on the other side of the integrated circuit. The software developer's solution was simple - don't use that data pattern. :-) (We found a power supply without the noise for that particular customer and redesigned the printed circuit board to eliminate the problem.)

DDJ: How would you describe your testing philosophy?
DH: I tell people, "All's fair in love, war, and software testing." By that I mean that if we testers can cause the software to act up, it almost doesn't matter how we caused it to happen. Except in extreme circumstance, if we can do it, it can happen in the field.

So what if it's a corner case; the software misbehaved.

Don't tell me a user wouldn't do that. I'm a user. I just did that. The software misbehaved.

It [almost] doesn't matter how we expose a defect. It's a defect. I report the behavior I observe. Then management can make an informed decision about whether or not anything should be done about it. These are entirely different questions.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DDH: Software testing isn't about pounding keys or acting like a user. Good software testing is hard work and technically challenging.

Excellent testing of software is far more challenging (from an engineering standpoint) than merely developing a program. The developer may have a difficult job converting the model we call requirements into a working computer program. They have to figure out one way to make it work. But, the the tester must think of all the various ways the program might fail, then select the most important of these, and *then* figure out how to exercise the program in such a way that we could tell whether or not the program has that defect. (Oh, by the way, we've now created the requirement for the test - we still have to develop it; a program development process in itself.)

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DH: I think the biggest challenge is overcoming the momentum from poor industry practices and misapplied standards. Today, we are using the same test approaches and techniques created in the 1960s and 1970s, and we're failing to test software well. It shouldn't be a surprise that the failed techniques created back then are failing still. But, today, the same arguments are put forward that were espoused then. "Standards are great, so everyone can do the wrong thing equally badly." "Certification is necessary in the industry so everyone will do testing the same way I've always done it." "All software and industries are the same, so one set of standards and strong legal enforcement is all we need."

The biggest challenge will be slowing down the continual push for conformity long enough for fresh ideas and approaches to take root. Improvement requires change. Learning requires openness to ideas we don't already have.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink