Five Questions With Michael Bolton

I first met Michael Bolton several years ago when he and I both happened to be in San Jose and he invited me to a meeting of the Weinborg (i.e., students of Jerry Weinberg). Since then he has become something of a mentor to me as I attempt to wrap my head around this Exploratory Testing thing he is always running on about. Michael is a frequent speaker at testing and software development conferences (including, natch, Software Development West), teaches Rapid Software Testing all over the world (my review here), and is an active participant on every testing-related discussion list I know. He has that seemingly rare knack of asking questions which make you think without making you feel stupid or put down - which I appreciate as I find myself pondering his questions a lot! Here is what Michael has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MB: Wow, that's a question with a lot of potential answers. To start with, I was brought up in a family where asking questions and seeking answers was encouraged. In that sense, I like to think that I've been testing all my life. When I started programming in the late 80s, I couldn't really separate the act of running my code and testing it; every time I believed something worked, there was a little daemon in the back of my head asking me "How do you know it works?" When I started working at Quarterdeck in the early 90s, I was in technical support. Our products were designed to provide memory management and multitasking for regular DOS applications, and there were zillions of those. Figuring out and reporting on why our products sometimes failed to achieve the expected result for our customers was very much a testing activity. In addition to that, everyone in technical support and most of the other people throughout the company tested our products as soon as the developers released builds.

My first formal job called "testing" was at Quarterdeck, co-ordinating testers and test activities for DESQview/X, QEMM, and DESQview. In that role I did a good deal of testing myself, partly to find problems on my own, and partly to reproduce and confirm reports that were coming into the development department. At some point during that period, I was given a copy of Cem Kaner's Testing Computer Software to read, and I devoured it; it was the best piece of technical writing I had ever seen. Later I became a program manager. The way I approached the job, it exemplified James Bach's definition of testing--"questioning a product in order to evaluate it". In fact, I found that it wasn't enough to question the product; I also had to question all of the systems around the product--the development team, the installation team, build and release control, technical support, marketing, and so forth. In those areas, the great performers all had that questioning attitude too, and the result was some pretty amazing software for its time.

In each of these roles, I developed a greater appreciation for the richness and complexity of testing. The unifying theme is the process of trying to make sense of things by asking questions and seeking answers.

DDJ: How would you describe your testing philosophy?
MB: My testing philosophy is closely aligned with Cem Kaner and James Bach, and it has been profoundly influenced by them, and by Jerry Weinberg. It's centered around the idea that testing is a process of investigation, finding out how stuff works in an open-ended search for information. That leaves a lot of room for testers both to generalize and to specialize. It suggests that testers need not, should not, all have the same skills. It means that all testers can bring new perspective and insight based on their own unique knowledge and experience. It also means that testers should remember to interact with the application as its intended user does. It's fun--and it might even be important--to do lots of clever automation stuff, to make plans, to draw diagrams, to write scripts and so on--but those things aren't the testing effort--they're tools that serve the testing effort. Over the last few years, the focus has drifted from "test everything" (which is impossible anyway) to "automate everything" (which is not only is impossible, but which may not serve the testing effort particularly well). I worry a lot about that; it feels to me like replacing one silver bullet with another.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
MB: I like the fact that you've divided the question into parts, because I think the answers are quite distinct. To me, the most important thing is that there are plenty of important things, and it might not be such a great an idea to settle on only one.

My answer today for testers: the most important thing is that testing has a lot of facets, but it's all being done for people--lots of people. We're providing information to management about the state of the product; we're assisting the developers in identifying risks and problems that could reduce the value of their work; we're trying to anticipate the problems that users will encounter in the product. Those different missions, for those different groups, mean that we can use lots of different approaches to find important problems. It's important to work with our managers and our project sponsors to negotiate our missions so that we can exploit and develop our skills in the ways that best serve the project.

Cem has recently been giving presentations in which he remarks that testing can't be hard science any more. In the old days, everybody who worked on the program could understand the whole thing--there were only a few thousand lines of code, a tractable number of variables. And to a great degree, product scope was limited enough that proving a program mathematically correct might have been possible and even useful. Systems today are too complex for that stuff. We can't hope to comprehend programs that have hundreds of thousands or millions of lines of code, never mind the interactions between them. We can't be sure of the reliability of the underlying libraries and operating systems, either. There are so many contexts in which a piece of software can be used that we have to settle for something more like a social science model, where there are enormous numbers of free variables, where answers aren't definitive, and where subjectivity of the analysis is a given. In that kind of realm, we have to accept the fact that we can't get certainty, since a changing context can change everything we think we know about the product. On the other hand, we can still do excellent work when we give people information that they value. Cem referred to the social sciences as giving us "partial answers that might be useful"--and I think that's bang-on. To me, testing really is like that, and I think it's important for testers to embrace the idea--and to help our colleagues in other roles to understand it, too.

For developers, there are a few things that are worth knowing about testing. The first is that we testers are trying hard to make you--and the rest of the project team--look good. If we sometimes seem a bit negative, it's because we're often focused on risk, where developer-folk are focused on the positive outcome. Those differences in mindset are natural and valuable, and both are necessary if we want to know how the product can work and how it might fail. The second point is that developers can really help us testers by building products that are more testable--by which we mostly mean controllability and visibility. It helps to speed testing enormously if we can throw some automated tests at a scriptable interface, and if we can obtain a detailed log of stuff that the application is doing. A third point for developers is that the more bugs we testers find, the worse our test coverage gets. It takes some amount of time to perform a test, but it takes a good deal longer to investigate and report a bug. That leads to a paradox: the more bugs we find, the less time we have to find bugs, because the time that we spend on investigation and reporting is time that we can't spend on the design and execution of new tests. That leads to the fourth thing, which is that unit testing--typically in a test- or feature- or behaviour-driven development model--can really help to get some of the trivial stuff out of the way. To the extent that I'm doing TDD myself, I'm finding that it really helps to make my code more robust while improving my programming speed. Jerry Weinberg told me once that in the old days, programmers needed to write lots of unit tests because the machinery was so unreliable; now that the machinery is really robust, the operating environments and programming languages and runtime libraries are hideously and complex, so there's a lot of risk there. Unit tests address that risk really early in the process, which keeps feedback loops nice and tight. In addition to that, they're a super way to do a great deal of regression testing--right at the source level.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MB: Oh, there's lots of stuff--"mythodology"--that I think we should be able to put to bed for a long, long nap. The first is the idea that we can use words like "always" or "never", or any other absolute language with reference to testing. One of the critical thinking games that James Bach and I like to play is called "Don't, Because, Instead". The object of the game is to take some practice that sounds like a good thing, give a reason why one shouldn't use it, and offer a new practice to use instead. The game works best when there's an even number of players and an odd number of practices, such that after a few rounds, you make a powerful argument in favour of the practice that you argued against in an earlier round. The game is designed to emphasize the potential value of each practice, and the foolishness of any claim that there's such a thing as a universal best practice. Context-driven testers try hard not to believe that--even to the extent that we recognize there are contexts in which context-driven thinking itself might not be such a hot idea.

Another thing that I would like to see go away is the obsession that some of us seem to have with numerical measurements of things to do with testing, especially with evaluation of the test effort with estimation. I'd prefer to see a focus on qualitative (and qualified) assesments instead. A lot of people who speak at conferences and write in magazines and books present these really elaborate mathematical models, but when you look at the way the numbers are calculated, there are so many assumptions, so many free variables, so much bogus precision that can be undermined by a tiny change in circumstances. Jon Bach and James are focusing on the notion of the testing story, and how it aligns with the product story, as being much more important. Numbers without a story attached are abstractions by definition, data without information.

When the project sponsors ask you for an estimate in how long it will take to test something, I think they're asking a question that doesn't have a good answer. There's an infinite number of tests that you could run, so you could take forever. That won't fly with the managers. You don't want to stop testing before the product ships, either. So here's one way to estimate the testing effort on a project: In my experience, the project owners have a ship date in mind anyway, so tell them that you'll test the bejeebers out of the product until they want to release it--and that you'll provide them with all the information that you possibly can until then. This approach isn't precise, but it's 100% accurate--and it's quick.

I think the current certification fad can and should be ignored until the certifiers actually watch you testing something and put some effort into examining your thought processes. Bugs don't follow bodies of knowledge, and they don't fill out multiple choice tests; they're often far subtler than that. The testing mindset has to be at least as subtle. Finding bugs, reporting information clearly, developing testing strategies, adapting to changing situations, understanding that nomenclature can vary from one context to another--those are only a few things that a tester has to be able to do well. A certification process that doesn't test for that isn't really certifying testers; it's certifying clerks. One of the people from one of the certifying organizations said to me once that he felt his company's certification should be viewed more as a learner's permit than as a real evaluation of expertise. I suddenly developed a lot more respect for him; I just wish that his company's marketing materials were as forthright as he was.

DDJ: Anything else I should ask you about?
MB: Ah--that's a question that I think you learned to ask from Jerry Weinberg. You could ask me about where testers might start a serious, advanced study on testing and on software quality--and my suggestion is to start by reading Jerry's stuff (his "author name" is Gerald M. Weinberg), joining the SHAPE Forum, and attending the AYE Conference. You could also ask how to get in touch with me so that I can answer more questions--an offer that our community tries to extend to everyone. My Web site is at http://www.developsense.com, and I love talking about testing in pretty much any available medium. Please feel free to drop me a line any time!

Posted by The Braidy Tester at 07:30 AM  Permalink