THE BOOK OF TESTING

Thoughts From a Braidy Tester

September 2007

J. B. Rainsberger is likely no stranger if you are a unit-testing Java developer. J.B. wrote the ever popular JUnit Recipes: Practical Methods for Programmer Testing, which has more-or-less become the Bible of Java development. He is a frequent contributor on the JUnit mailing lists and conferences around the world, and he founded XPDay North America conference. These are some of the reasons he was awarded the Agile Alliance's 2005 Gordon Pask Award for Contributions to Agile Practice. Here is what Joe has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JBR: I wrote test plans for a project in my Software Engineering graduate class in 1998. This mostly consisted of writing out manual test scripts in English, which seems quaint now. My impression was that rote testing needs to be automated.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JBR: Testers in large organizations spend enormous amounts of time executing pointless tests manually. What a waste.

DDJ: What is the most interesting bug you have seen?
JBR: While at IBM, I saw a bug that took 3 weeks to fix. The application in question was consuming native graphics resources in Windows until Windows was no longer capable of painting a window. Fortunately, I wasn't the one chasing it; but I felt pretty bad for the one who did.

DDJ: How would you describe your testing philosophy?
JBR: Automate all the rote tests, but keep an experienced exploratory tester involved in the project. Test-drive every line of code you reasonably can.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JBR: Testers are not there to wipe the programmer's arse. The programmer should not deliver to testers stupid defects they could reasonably find and fix.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JBR: Writing test plans. Instead, execute a test, then consider automating it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JBR: As more programmers test-drive code, testers will need to develop real testing skill, and not just execute on someone else's test plan.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JBR: Ask me, "Why do you test-drive your code?"

My answer, "Because I like the resulting designs. Validation is a nice side-effect."

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

John N. D'Addamio has a bit of experience in the computing world...thirty-five years of it or so. Once upon a time he was working towards a PhD on operating system security. He never finished, though, because he decided that his subject was in fact an oxymoron. So he left academia and found a series of positions which recompensed much better than a doctorate ever would have. Positions liking developing in assembly on IBM 360 and 370s. Positions like developing, testing, and leading programming efforts on Unix. And, most recently, positions like testing for Microsoft, where he tests tools which Microsoft plans to release as open source.

Here is what John has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JND: Every programmer’s first introduction to testing is running the code they wrote for class assignments! I was lucky enough to have an instructor who talked about the basics of testing and debugging along with teaching the language. At the time, I thought testing was more of an art than a science. I also thought similarly about coding and debugging. While some people are more adept at one of the three tasks than others, there is now enough good information that any professional programmer should be able to do a good job at all aspects of our job.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JND: The relatively poor set of test tools available for automated testing of GUI applications is surprising. Whether you are talking about web applications with a client-server model or locally installed applications, there is not much that is widely available either commercially, free binary only, or open source.

DDJ: How would you describe your testing philosophy?
JND:

1. Plan a minimal number of tests necessary to cover all the application’s functional requirements. Ideally, these requirements should be in a spec but, all too often, the tester has to pull them together from emails, meeting notes, etc.


2. Prioritize the implementation of tests so that the most important get done first.


3. Automate as many tests as possible given the testing tools available to you.


4. After all the application’s functional requirements have been tested, use code coverage tools to identify untested code in the application. You will usually find a combination of gaps in the test suite and some unnecessary application code.

DDJ: What do you think is the most important thing for a tester to know?
JND: Testers have to know how to work with developers in a collaborative rather than an adversarial way.

DDJ: To do?
JND: Pay attention to details and requirement combinations! Of course, we have to keep the “big picture” in mind while testing but the details of behavioral requirements (especially in combination) are usually where you find problems.

DDJ: For developers to know and do about testing?
JND: Developers have to be able to write good unit tests. Two reasons: good unit tests will eliminate a lot of bugs before the test team gets the application and good unit tests allow the test team to concentrate on integration and scenario level tests. The net result of both effects is the team turns out a higher quality product if the developers provide unit tests along with their application code. Obviously, test driven development can provide huge advantages for both the developers and testers. The difficulty is often getting management to accept the idea because they see a longer development schedule than they would like.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JDN: I think the biggest challenge for the test discipline is the same as the single biggest challenge for software industry in general. We have to find a balance of planning, documentation, development and stabilization (including testing and bug fixes) that makes sense. Large segments of the industry spend very little time on planning. We often just start writing code at a vague or moving set of requirements and then spend most of our time fixing bugs and rewriting the code to meet the unexpected requirements. The technical documentation needed for maintenance is usually neglected. On the other extreme, we have projects that are written for the federal government including the various branches of the military. These projects usually have such rigid planning and documentation requirements that senior programmers spend as much as two thirds of their time on the project doing planning and documentation. Of course, this shortens the coding and stabilization phases as well as providing good documentation for maintenance efforts. Neither model is ideal for the software industry at large so our challenge is to find a balance that actually works well.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Eric Sink has been on my Must Read list for years now. His series on the business of software for MSDN - now collected into a most excellent book titled, appropriately enough, Eric Sink on the Business of Software - is the place to start; if you like what you read, head on over to his Eric.Weblog() for much, much more. Or visit Joel On Software's Business Of Software forum, which Eric moderates.

Back in the day Eric founded the AbiWord project and was a project lead at Spyglass when they were designing the original versions of the Browser-Currently-Known-As-Internet-Explorer. Today he is a Software Craftsman at SourceGear. Here is what Eric has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
ES: I think I am continually surprised how hard testing is.

One goal of testing is to find (and fix) bugs before our customers do. Our products go out to tens of thousands of users. That's considered a very, very small customer base when you remember that Microsoft's products go out to hundreds of millions of users. Still, even at our scale, those users represent an astonishing diversity of configurations and usage styles.

Just to use round numbers, think of the problem like this: You have a team of ten people and their computers. They are trying to find all the bugs that would be found by a group of ten thousand people and their computers. The only advantage is that the former group is specifically looking for the bugs while the latter group is specifically hoping not to encounter them.

It's an amazingly hard problem.

DDJ: What is the most interesting bug you have seen?
ES: My favorite bug happened early in my career while working on scientific visualization tools at Spyglass.

It all started when a user called our tech support team and said, "I was using your product and MPW crashed." (MPW stands for Macintosh Programmers Workshop, Apple's development IDE back in that day.)

Our first reaction was to assume that MPW's crash had nothing to do with us. I mean really. Why blame us because MPW crashed? He was using lots of other things at the same time. Did he call his electric company and tell them he was using the power grid when MPW crashed?

So we asked all the usual questions. Does MPW always crash when using our product? No. How often does it happen? I don't know.

We concluded that there was no connection. Our customers were mostly scientists. We were accustomed to getting some very strange phone calls.

But this particular situation happened again with somebody else. And then somebody else. And it kept happening. Eventually we had to conclude that the problem was real. But since it was completely non-reproducible, we had no idea how to fix it.

I still remember my manager boasting that if anyone could ever make the bug reproducible, he would fix it that day. And one day, that's exactly what happened. A user called in with a recipe to reproduce the bug. It had to be followed exactly, and it had to be done on a certain kind of Mac. At the end of the sequence of steps, boom, MPW crashed.

And my manager found and fixed the bug within a couple hours. We were forgetting to free a palette handle.

DDJ: How would you describe your testing philosophy?
ES: My philosophy about testing is all about balance. Some kinds of testing seem to produce more bang for the buck than others, but the best results happen when multiple kinds of testing are used. Testing is not an area where it pays to put all your eggs in one basket.

My own particular interest is in automated testing, especially when used with code coverage. My pet peeve is people who argue against automated testing by saying that it's not enough. For some reason, people get themselves in the mindset that automated testing is only worth the trouble if it becomes sufficient for all their testing needs. Well of course automated testing is not enough! Nobody said it was or should be, but there are darn good reasons for doing it anyway. It's all about balance.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ES: I personally think everything works better when we all agree that the job of testing is not to ensure or verify the quality of the software, but rather, to measure the quality of the software. This is a subtle distinction, but an important one.

In the course of shipping a product, good testing will find lots of "bugs" that will not be fixed. I argued this view in an article on my blog entitled "My Life as a Code Economist". For anyone with real experience shipping a product, this is obvious. For anyone else, this is heretical.

We don't want testers to make sure there are no bugs. We want testers to tell us about the bugs that we've got so we can make an informed and wise decision about what to do with them.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ES: Bug tracking tools are growing up, and fast. We have traditionally thought of bug-tracking as a very simple tool. However, bug-tracking tools are getting more and more integrated with other team development tools like version control and builds. The rate of change in the toolset promises to keep our lives interesting for some time to come.

Posted by The Braidy Tester at 07:30 AM  Permalink |