THE BOOK OF TESTING

Thoughts From a Braidy Tester

October 2007

Kent Beck is likely a familiar name to you, especially if you are at all familiar with Agile development. I first heard about him in the context of Extreme Programming, which concept blew my mind. Then I read Test-Driven Development and Refactoring, which radically changed the way I wrote software. (You mean there are processes I can follow whereby all this code I write is highly likely to write the first time, and all these changes I make to remove code smells are much less likely to break something? Sign me up!) Long time readers of DDJ may remember Kent's articles on the care and feeding of Smalltalk.

Today Kent runs his Three River Institute, where he teaches people about everything I've mentioned and more. Here is what Kent has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KB: I first saw real testing when I worked at MasPar Computer with Christopher Glaeser (the person behind the Nullstone compiler optimization tests). He wrote five lines of test code for every line of compiler he wrote. His code was rock solid but he was also very productive. I had never seen a programmer write extensive tests before, so it was a real eye opener for me.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KB: When I went to school, testers were looked down on as second class citizens. I think what surprised me most about testing was how interesting, challenging, and productive an activity it was.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
KB: The key is to make testing more and more part of the inner loop of development, so architecture, design, implementation, and performance decisions can be subjected to immediate experiment. All the feedback we're currently willing to wait for needs to be available sooner.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
KB: In both cases the most important thing to know and do is communicate with people, even when those people have very different perspectives. Testing is a form of unambiguous communication: "The system should do this." However, around that unambiguous communication are all sorts of typically human communications, with all the ambiguity, contradiction, and emotion that implies.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KB: Perfection was emphasized when I learned about testing in school. The intractability of exhaustive testing was held up as an excuse not to test at all. I think that's misguided. Organizations can function just fine in the presence of software defects. Getting to that level of defects economically is the challenge (hence the emphasis in XP on identifying and fixing defects within minutes or hours). That said, I think a fairly simple set of technical and social practices can eliminate the majority of current defects. When software runs ten times more reliably, organizations will begin to count on that, which will create the platform to improve software quality further.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Tom Gallagher got into security testing in perhaps the best way imaginable: by compromising an ISP's computer system! (With their permission, he says....) He joined Microsoft out of college and has been penetration (white-hat) testing Office ever since. These days he leads Office's security test team, continues to pen test, writes security testing tools, and educates folks in all things security. You might have seen him speaking at the UNCC Fall Cyber Security Symposium a few weeks ago or at last year's Black Hat USA security conference; you might also have noticed his recent (co-authored) book Hunting Security Bugs at your local bookstore. And, if you're into browsing notices of security defects, you likely have seen his name connected to various of those reports.

Here is what Tom has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
TG: Before I was formally introduced to testing, I was trying to explore and break things. I later found out I was really testing. At an early age I took a particular liking to automated phone systems. There are a lot more options than the automated voice tells you about! Even the dial tone has many more options than simply calling a phone number. In the beginning, I randomly tried things. As I spent more time I saw the need to be organized so I could be more efficient and reproduce things more reliably. I found much of the testing very enjoyable and like solving a puzzle or playing a game.

My first programming teacher used really good test cases to validate programming assignments. He didn’t tell the class his test input values ahead of time. This forced me to spend a lot of time thinking carefully about how things might go wrong. I knew other teachers used mostly expected values as input when grading assignments. At the time it seemed my teacher was just evil, but this forced me to test my programs more thoroughly.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
TG: The many different areas of testing. I had no idea how much work is involved to properly test an area. Things like accessibility and international testing were not immediately apparent to me. As I learn more, there continues to be more things involved in testing.

DDJ: What is the most interesting bug you have seen?
TG: If I told you, I’d have to kill you. =) Most of the interesting bugs I’ve seen are security issues that aren’t publicly known. I shouldn’t really talk about the details.

Since I can’t share the technical details, I’ll share the story around one of the bugs. Robert Fly and I were brainstorming around a new attack which might allow us to compromise the client application we were testing. We thought our idea was decent so we started looking for places in the application that might be vulnerable. After a little bit of time, I found a place that was vulnerable and was ready to log the bug. Luckily Microsoft takes security seriously, but I thought I might have to sell this bug a little. Tell a developer they have a buffer overrun, they get it and fix it. Tell someone they are vulnerable to a new attack, sometimes you have some explaining to do. I also personally enjoy writing exploit code so I thought I would do it in this case. My bug’s repro steps were something along the lines of “Visit this URL. Notice your name and annual salary are accessed and displayed by an attacker’s code.” We were at a time in the product cycle when all bugs/fixes requires heavy review (Microsoft’s triage process). First, individuals review the bug privately and then meet to discuss the issue. This meeting typically involves a room of feature owners and senior level developers, testers, and program managers reviewing the proposed change and if it is necessary. The bug details are displayed on big projection screen. I was not present for the review meeting, but the bug was deemed a showstopper for the product. I was also told during the meeting the presenter refused to visit my URL because he didn’t want his personal information displayed to the rest of the attendees. My proof of concept code seemed to sell the importance of the issue.

Recent research around exploiting uninitialized memory bugs is also very interesting to me.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
TG: In my opinion, it is essential to understand the technologies/languages leveraged by the product being tested and study the product’s implementation. I see a lot of testing happening from the user’s perspective by only using the product as a black box. Software isn’t really a black box. Source code is a valuable resource if available. Even when no source code is available, the software product being tested can tell you a lot about how it really works. Testing while using debugging and monitoring tools like Process Explorer, a packet sniffer, an API monitor, etc. are very helpful but often overlooked. With knowledge of how the product actually works, it is easier to identify places where bugs are likely to occur.

It is helpful if developers help testers by identifying areas of concern and explaining how the code works. Some of the better developers I know work closely with testers to ensure shipping quality code. These developers also understand what the testers plan to test, the tools that are being used, weaknesses that might be present in the plan or tools, and is able to assist the testers strengthen their approach.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
TG: Security. Feature level testers understand the way the product actually works because they use the product and try so many different things. With this knowledge feature level testers are in a good position to identify security bugs. Not enough feature testers include deep security testing as part of their testing. Often security testing isn’t easy. Many times the product behaves as intended, but there is still a security issue. Many feature testers haven’t had lots of training around security testing. I worry that if this doesn’t change we will continue to have many serious security bugs in critical pieces of software.

Just so I don’t sound all doom and gloom here, the team I work on at Microsoft attempts to solve this problem by training feature testers about security testing and requires testers to security test all of their features. This is also the focus of the book entitled Hunting Security Bugs from Microsoft Press.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

I am excited to announce that as of now Microsoft's MSDN has a Tester Center! Here you will find articles, blogs, videos, the Software Testing Discussion Forum, plus an aggregation of all the testing-related Microsoftie-run blogs the Tester Central folks know about. Head on over for all sorts of testing goodness!

Posted by The Braidy Tester at 12:05 PM  Permalink |

Douglas Hoffman owns and operates Software Quality Methods, where he does test planning, managing, and training for companies such as Hewlett-Packard, Palm Computing, and Sun Microsystems. You may have seen him talk at PNSQC last week, which is the latest of many papers and presentations he has given over the last many years. Douglas teaches testing at the University of California at Santa Cruz Extension, is a founding member of the Association for Software Testing, and is a Fellow of the American Society for Quality. Here is what Douglas has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DH: This is hard to answer. Besides being a long time ago (I started programming using coding sheets and 36 hour turn-around time from submission to getting the printout - I didn't even see any punch cards), my introduction to testing was slow immersion over 8 or so years as a developer. It wasn't until the early 70's that I did any work that I thought of as being testing, and then it was system level testing on a computer system manufacturing floor.

I wrote the production instructions for technicians to assemble and test customized minicomputer systems. Because my audience was production technicians, the tests had clear pass/fail criteria. I learned something back then that is still absent in some software testing - there has to be some way of determining a verdict when the test results come back. There has to be some way to tell pass from fail. A test needs to show whether or not the software behaved well. Someone also needs to care about whether or not it worked - tests aren't helpful when, if they fail, nobody is going to do anything. That just seems like a frustrating waste of time.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
DH: I'm still surprised and disappointed by how may managers and testers think testing improves quality. There seems to be a perpetual myth that just knowing about a problem fixes it. "Why didn't Testing find that defect?" isn't the first or most important question to ask. Testing didn't create the defect. Testing can't find all the defects. Testing doesn't take them out. The information provided by testing is critical for understanding the quality and to avoid too many embarrassments. But, defects are created before the program got into testing. That's where the most important questions are.

DDJ: What is the most interesting bug you have seen?
DH: I've been fascinated so many ways by so many defects. How simply and elegantly one incorrect bit in the code can so thoroughly trash the work done by the other 100 Megabytes of the program. Which is the most interesting? The comment change that shut down the 100+ developers for a week. The rushed, untested, one line fix that the VP forced into production (to shave a week off of the release) that overnight rendered 16,000 desktop systems useless (until the technicians could get around to individually reloaded them). The square root problem that caused just two of the 4 Billion computations to be wrong.

Probably at the top of my list is a data-dependent DMA error. (Direct Memory Access was a hardware port that sent or received blocks of data without tying up the CPU. It was used especially with disk and tape.) The problem was due to a combination of network speed, printed circuit board layout, power supply noise, and specific data values that led to different bits arriving at the disk drive than were in the memory. I spent a week with my head plastered against an oscilloscope hood, looking at bits fly by. I would tweak the data, look at the bits. Probe the next data line, probe on the other side of the integrated circuit. The software developer's solution was simple - don't use that data pattern. :-) (We found a power supply without the noise for that particular customer and redesigned the printed circuit board to eliminate the problem.)

DDJ: How would you describe your testing philosophy?
DH: I tell people, "All's fair in love, war, and software testing." By that I mean that if we testers can cause the software to act up, it almost doesn't matter how we caused it to happen. Except in extreme circumstance, if we can do it, it can happen in the field.

So what if it's a corner case; the software misbehaved.

Don't tell me a user wouldn't do that. I'm a user. I just did that. The software misbehaved.

It [almost] doesn't matter how we expose a defect. It's a defect. I report the behavior I observe. Then management can make an informed decision about whether or not anything should be done about it. These are entirely different questions.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DDH: Software testing isn't about pounding keys or acting like a user. Good software testing is hard work and technically challenging.

Excellent testing of software is far more challenging (from an engineering standpoint) than merely developing a program. The developer may have a difficult job converting the model we call requirements into a working computer program. They have to figure out one way to make it work. But, the the tester must think of all the various ways the program might fail, then select the most important of these, and *then* figure out how to exercise the program in such a way that we could tell whether or not the program has that defect. (Oh, by the way, we've now created the requirement for the test - we still have to develop it; a program development process in itself.)

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DH: I think the biggest challenge is overcoming the momentum from poor industry practices and misapplied standards. Today, we are using the same test approaches and techniques created in the 1960s and 1970s, and we're failing to test software well. It shouldn't be a surprise that the failed techniques created back then are failing still. But, today, the same arguments are put forward that were espoused then. "Standards are great, so everyone can do the wrong thing equally badly." "Certification is necessary in the industry so everyone will do testing the same way I've always done it." "All software and industries are the same, so one set of standards and strong legal enforcement is all we need."

The biggest challenge will be slowing down the continual push for conformity long enough for fresh ideas and approaches to take root. Improvement requires change. Learning requires openness to ideas we don't already have.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Lyne Brown is currently a Development Lead for Microsoft's Macintosh Business Unit tools+automation+build team, which means she spends a lot of time working with testers to build tools that will make their jobs easier, faster, and even more fun. For thirteen years she has been doing this, through many a title, management, and organization change. Before that she helped engineers at the GM Proving Grounds do their job, and before that she was a CS student after deciding that ornithology wasn't really what she wanted to do with her life.

Here is what Lyne has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
LB: My first introduction to testing was when I did a phone interview with Paul Williams. I sent in an unsolicited resume to Microsoft in ’94 and Paul called me with an SDE/T position that was available in the new Office core team. I didn’t realize at the time that testing was a possible career path. At first I was deterred but the more we chatted the more it piqued my interest and the more I realized that I might actually have an aptitude for testing. I had always had an interest in how things worked and how I could break them :).

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
LB: I was initially surprised at just how rich of a career path that testing can offer. Testing has changed substantially over the past 13 years that I have been at Microsoft. There is always something new in the wind. Whether it is richer automation, model-based testing, PICT, white box testing, a well-defined automation stack that leads to such concepts as decoupled automation, etc. – there is always something new to learn that can be another tool in the tester’s arsenal.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
LB: Testers: 1) don’t become an automaton – keep that child-like curiosity and desire to “break” things as you discover their inner workings. 2) Look at the code you are testing whenever possible. If you can look at the code from the eyes of a tester you can better deduce how to test it as well as feed those observations back to the dev. Developers: 1) work closely with your tester. Understand that there is no hierarchy in the engineering disciplines. You are all working together for one common cause – delighting our customers. 2) Give the testers what they need to help them better test the product. If a certain feature does not have an exposed API, then make sure you expose it thru accessibility.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
LB: I think the biggest challenge will be in balancing the growth in technical areas (automation, white box testing, etc.) with good old-fashioned testing skills. I see all these sharp, new, technically sound testers being hired but they don’t seem to have the rudimentary skills (ad hoc-ability, curiosity, thinking outside the box) needed to really be a great tester. I believe there needs to be more education done in teaching testers the skill set they need to be great ad-hoc testers and there also needs to be more time dedicated in the product life cycle for ad-hoc testing. Where did all the bug bashes of yore go?

DDJ: Is there anything else you would like to say?
LB: I would just like to state for the record that the comments above are my observations from outside the role of a tester. I only owned a feature (Office toolbars) for one month before it was decided that our team would work on white-box testing tools and the automation system and so I handed my feature to someone else. All my 13 years at Microsoft have been in the role of an individual contributor or lead for a team that is dedicated to tools development, primarily for the test organization. I have been in the same role the entire time but have been titled as SDE/T, SDET and SDE and have been in both the test and now the dev organization. I am giving this background information because my hope is that someday the distinction between people in the engineering disciplines is not in their title but instead in their role. This may become more of a reality as more teams move to agile-type methodologies such as SCRUM and Feature Crews. I see that the unspoken caste system that has been around at Microsoft in the engineering world for as long as I can remember is slowly disintegrating but it still exists at least in the minds of some of the old-timers.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |