Testing & Debugging Blog

This post will be my last. For now at least - I cannot see into the future, so I will not say I will never blog here again.

Thanks to all of you readers, and an especial thanks to each of you who commented on my writings. See you 'round the blogosphere!

Five Questions With Dave Liebreich

Dave Liebreich bills himself as a software testing architect, a consultant, a coach, and a manager. I bill him as someone who knows a whole lot about testing. Regardless of whether he is helping your team as an individual contributor, your Director Of Test, or anything in between, Dave can help your team become its Best Option version of yourself.

Dave has served as Test Architect at Mozilla Corporation and as QA Manager at Yahoo!, so he knows his way around complex, distributed, networked systems. What's more, he has the know-how to translate his experiences into the product you are building, even if said product is neither complex nor distributed nor networked. Through it all, Dave helps you keep track of what you are doing, why you are doing it, and how you will know you are done.

Here is what Dave has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DL: First introduction: Back in the days before they called it I.T., I was a system administrator for a communications infrastructure equipment maker. I moved from the sysadmin group to the test group, and started writing automated test libraries. After a while, I was assigned some testing responsibilities.

Left me thinking: Mostly due to dumb luck, I realised I was testing as a sysadmin, not as a tester. I noticed that there were some testers who had a 'knack' for quickly finding problems in new builds and with new products. I wanted to be like that.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
DL: Testing is part of any creative work - everyone asks technical questions of the technical stuff they do. The idea of having specific skills and studies in testing was a bit of a revelation.

DDJ: What is the most interesting bug you have seen?
DL: As a user? I worked as a bank teller during college. One day, I noticed I had created $500.

As an observer of another tester? I saw someone find a memory leak caused by moving the pointer across the UI.

As a tester? I wrote a script to populate a network management system with dummy nodes. The script printed out the name of each node as it was created. I left the shell window open and noticed that the script slowed down as more and more nodes were created.

As a professional? The next one.

DDJ: How would you describe your testing philosophy?
DL: Testing is

answering questions

shortening feedback loops to answer questions faster

moving feedback earlier in the process to answer questions sooner

asking questions like "given what we just learned, what kinds of improvements to what we do can and should we consider?" so that we can ask and answer better questions

starting with each tester's strengths in noticing stuff, use critical thinking techniques to notice more stuff, so we can discover questions we should have asked, but didn't

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DL: Most important for a tester to know: applied critical thinking skills. Most important to do: apply them to both the problem domain and the solution domain.

For developers to know and do: ask testers to help find something developers would want to know, but do not currently know, about the system under test. Not just developers - same for dev managers, project managers, customers, stakeholders, etc.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Scott Louvau

Scott Louvau writes software for Microsoft. He doesn't write any software you have ever used, though, unless you have ever worked on his team. The software Scott writes, you see, is the software his team uses to test their product. To paraphrase the commercial, Scott doesn't test the software you use, he makes the testing of the software you use better.

Scott tells me that if he was not in software he would be some sort of craftsman because "I really like the idea of doing a job you can get better at every day". He recounts a Discovery Channel special on how aircraft carriers are built, a process which involves welding together giant panels of steel. The host watched as a welder took a panel which had buckled from the heat of welding it and completely flattened it with a single hit of his hammer. When the host asked the welder how he knew where and how to hit the panel, the welder replied "Twenty-eight years on the job".

Although Scott has only six years on the job so far, I am sure his teammates would agree that he knows where and how to hit their buckled panels in order to flatten them out. Here is what Scott has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SL: Actually, my introduction to testing was an introduction to not testing. I wrote my first database application for my first real job, and excitedly told my boss it was ready to be released. Within two minutes he'd found a bug and within an hour one bad enough that it wasn't usable by customers.

I learned that if you haven't tested it, it doesn't work. If you wonder if something works and try it and it does, the odds are very good that it works because someone earlier wondered the same thing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SL: I guess it's that testing is so easy, and yet so difficult. It's very easy to break something and to think of the basic things about a system that should work and that you should try. It's not so easy to quickly find bad bugs in a relatively good product and explain why they are important. It's incredibly hard to choose the right tests from all of the possibilities and get them running more cheaply than the code they test.

DDJ: What is the most interesting bug you have seen?
SL: There are so many possible categories.

The worst bug I found caused Visual Studio to crash and delete as much of itself as it could.

The most entertaining bug I found had the offending developer's name burned into it. His My Documents folder became the default for all projects for anyone.

The subtlest bug I found was one where the arrow keys didn't work in one particular combobox in one dialog for some still unknown reason.

I can't sit down and look for bugs for more than half an hour without smiling. It's amazing how many little things there are to go wrong.

DDJ: How would you describe your testing philosophy?
SL: I guess I would say "to be pragmatic". Ultimately my goal is to get the most useful possible product out. I start with the things my product must be able to do and work my way into the soft spots I find as I go. I try to blend automating to prove the product works every day with exploring to find the areas that need more polish to really be useful to people. It's an interesting balance.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SL: I guess it's to remember that we all have the same overall goal - to produce the right product for our customers. I found the process of triage (deciding which bugs to fix and which not to) very frustrating until I really internalized the idea that we're building the same thing.

Remember that many users will have to use your product when you release it, and will cope with the bugs you've created for them every day. Remember that they are also dealing with not having your product every day it isn't there.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
SL: Metrics. They can be useful, but are so often abused to try to force things to be "done". Ultimately it's up to the people working on a project to really decide when it's done, and the metrics should only be potential tools to guide their efforts. If you would not feel good signing your name on the box of the product you've built, don't release it yet.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
SL: I think our hardest challenge will be to cope with the multi-core/many-core transition that's happening. Across computing I don't think we're great at writing massively multi-threaded code, testing it effectively, and reproducing and fixing defects. We will need to be.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
SL: Three questions:
What do you consider the key(s) to your success?
I think two things have served me very well. First, I don't like to just use something that works. I want to know how it works and why it works. Second, I try to do something better every day - automate something I've done manually, figure out a little code design problem - something. It adds up. =)

What is the most important thing you've learned as a tester?
The same thing I said for "What do you think is the most important thing for a tester to know" - we're all working toward the same goal.

Why do you like being a tester?
I think that software testing is much more of a "frontier" than development. We collectively know much more about how to write services and applications and utilities than software tests. I think testing is also much more open ended - anything you can think of to decide whether the code is good is fair game. I guess I'm also just fascinated to see my programs using other programs like a user would.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Michael Kelly

Michael Kelly is a tester's tester. He talks about testing on his blog as well as in magazines ranging from InformIT to Better Software. He talks about testing at the Indianapolis Workshops on Software Testing he co-founded as well as at conferences ranging from the Indianapolis Quality Assurance Association Quality Enrichment Conference to the upcoming Pacific Northwest Software Quality Conference. And he tests software for companies large and small as well as for the fun of it.

Regardless of how you come to know Michael, his passion for improving the craft of testing and for helping testers improve themselves comes shining through. Here is what Michael has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MDK: I started my career as an intern doing GUI-level test automation in a Fortune 500 company. In a matter of months I had found some of the works on that topic by Kaner, Pettichord, Marick, and others and was applying everything I could to the problems we were working to solve. After less than a year, myself and other $12-an-hour interns were teaching the $150-an-hour consultants how to write more maintainable tests and how they could get better coverage by making simple changes. The experience left me with an immediate sense of skepticism on the industry. I wondered at how those well-paid, well-meaning, and otherwise intelligent people could do work so poorly.

Once I entered the workforce proper, it didn't take me long to figure out testing would be my permanent home. It also didn't take me long to figure out what the problem had been. I found that most popular testing literature trivialized the work that we do and that people actually believed what tool vendors said. It was amazing to me that a group of twenty-year-olds could see through the marketing hype and experienced professionals couldn't. So, I sought out the only group of people I saw questioning the status quo. It was at the Austin Workshop on Test Automation that I eventually found the community that I could call home. They questioned and challenged everything, asked for empirical evidence of claims, and viewed testing as a cognitively-complex and interdisciplinary profession.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MDK: What has surprised me the most is how much I keep learning. I still feel like I did as an intern. There's too much to read, too much to try, not enough time to practice, and never enough time to collaborate with others. I know that I know more then I did ten years ago, but when I look at everything I still only have a passing understanding of, I feel overwhelmed! It's not simply a problem of keeping up with technology, it's also finding time to learn our craft's history and to explore what I can learn about testing from philosophy, art, mathematics, and business; a few of the places from where I've developed my understanding of what we do as testers.

DDJ: How would you describe your testing philosophy?
MDK: I identify myself as a member of the context-driven community, and find that much of my testing philosophy mirrors the work done by James Bach, Jon Bach, Michael Bolton, Jonathan Kohl, Cem Kaner, and others doing work in rapid testing and exploratory testing. I think most testers would be well served by developing a deeper understanding of scientific and systems thinking, heuristics and biases, and project and business management. I would also encourage them to explore areas of interest outside of software testing and to bring something back to the field from those interests. As I write this, we just finished the third conference for the Association for Software Testing and at the conference we had talks pulling lessons from music, magic, lacrosse, finance, improv comedy, statistics, and civil engineering.

DDJ: What is the most interesting bug you have seen?
MDK: I've written about this before, but I remember my first security bug. It was so simple, I stumbled over it accidentally. (Well, I told the very angry people who were upset with me that it was an accident.) The problem started with a developer who had left his or her user ID in a code comment on the login page for a production system. It looked something like this:

login	enter your username
password	enter your password

Note the comment enclosed in the tags. Out of curiosity, I entered the user ID (u2x34t) from the source code into the username field and tried to guess the password. I was rewarded with this:

"The password you entered is incorrect. Please try again."

I say rewarded because the first user ID I tried gave me this error:

"The user id you entered is not recognized. Please try again."

The specificity of the error messages for the system indicated that I was on the right track. I wouldn't have known that if the system had consistently displayed an error message similar to this one:

"The user id and/or password you entered are incorrect, please try again."

At this point, I knew I had a valid user ID, but I still didn't have the password. I didn't want to simply guess because I didn't want to lock the ID (earlier tests had shown that to be a problem). Instead, I started to wonder about that comment the developer made in the source code:

"Removed link to defect tracking system"

I asked myself some questions: What tracking system? How did they remove it? And where did it go? I looked at the source for more clues, but none could be found, at least to my untrained eye. I needed more source code. I figured that if the developer left comments on the login page, there was a good chance he or she left them in other code as well. At the bottom of the login page was a link to a help page. ("Need help logging in? Forgot your ID?") I followed that link and looked at the source for the help screen, where I found something similar to this:

To remove the link to the defect tracking system, the developer had simply commented out the link. Not only that�the link included a parameter for the user currently logged in so it would know who submitted the ticket. At that point, I had a URL that required a user ID, and I had a user ID. I simply copied the URL, pasted it into the address bar, typed the user ID in the appropriate place, and hit Enter. The system displayed an error page for a defect tracking system, stating that the system was no longer in use. I initially thought I had hit a dead end, but then I saw a link at the bottom of the page: "Return to application." I clicked the link and was rewarded with the home page (in production) for the application I was attempting to access. No password required!

After that, I was hooked, and I had to learn more about security testing.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MDK: I think there is a large focus on counting things in our industry. Many people count tests, use cases, defects, and even testers. We're obsessed with counting. I hear people say things like, "We have 4,000 tests" or "We found 50 defects" without any indication about what's actually being covered by the tests and what risks are addressed or what severity the defects are. There seems to be an impression that everything is equivalent or that if you just communicate a number you've communicated some meaning.

I don't find counting to be useful outside of its use as a general indicator for size or volume within a specific project context. On past projects, I've seen people count tests like they are lines of code. I find that you fall into all the same problems programmers run into when they do that. It may be a useful metric in some instances, but rarely is it a meaningful metric.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Dmitri Klementiev

Dmitri Klementiev tests software at Microsoft. No, he develops software for Microsoft. No, he tests software for Microsoft. I'm confused.

Oh, I get it: Dmitri is the type of tester for whom Microsoft originally coined the title Software Development Engineer in Test: he spends his days developing software to help other testers better do their job. In Dmitri's case this has largely meant putting massive amounts of time into developing the tool many Microsoft testers use to automate their drive-my-application's-user-interface test cases. Dmitri's Doctorate Of Mathematics and experience in mathematical and computer modeling seems to be holding him in good stead as he navigates the intricacies of automating the user interfaces of Microsoft's many different applications and operating systems.

Here is what Dmitri has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DK: When I joined Microsoft in June 1998. Automation that can be greatly improved and increase the efficiency of testing.

DDJ: How would you describe your testing philosophy?
DK: We have to deliver high quality products and our customers should love our products which means an elegant and "comfortable" (for users, not for us) design. Testing is a "quality gate keeping" in this process.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DK: Testers - their product, technologies that are used in the products, customer expectations, industry standards, other related products on the market. Programming skills to write automation. Should drive the quality of the products we ship.

Devs - how the products are tested: test plans, approaches, automation. Also know and support requirements to make the tester�s job (including automation) more efficient.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DK: Keep up with new technologies. Make automation much more efficient.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
DK: What�s the difference between a porcupine and a hedgehog. And I would answer "I don�t know" :)

DDJ: Is there anything else you would like to say?
DK: Since I am in UI automation world I would say that we are still pretty much in the beginning of the game - a lot of very complex and at the same time very interesting problems ahead. Let's be those who find good solutions for them.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Gil Broza

Gil Broza broke Edgar Allen Poe's second cipher. Gil Broza is a certified NLP Master Practitioner. Gil Broza is living what he describes as a "wild life" as he learns how to raise twins.

Perhaps more germane to this blog, however, is that Gil Broza has been working on software for close to two decades and has been XPing and Agileing for much of that time. Regardless of whether you have a financial, bioinformatics, or content delivery system to maintain or to build, Gil can show you, coach you, and help you do so in an Agile and Industrial XP fashion. You may have heard Gil talking at Agile 2008, at DDJ's own SD Best Practices conference, or at one of Industrial Logic's workshops on various XP-related topics.

Here is what Gil has to say:

]]> DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
GB: While many organizations prefer to hire intelligent, bright, hard-working testers, some companies appear to think that any warm body will do. Even after almost two decades in the software world, this mindset still amazes me. I guess it arises from the concept of the test plan, or from the notion that given a feature's spec and a working build, a person should really just need to "bang on it" for X hours to prove its correctness. As a firm believer in Exploratory Testing and think-once-then-automate, I think testers require careful hiring just like any other role.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
GB: A critical principle in software development, I believe, is the concept that "testing" and "tests" aren't about finding bugs. Rather, they ascertain the software's fitness to the customer's purpose and goals.

As an XP trainer and coach, I teach developers many practices that were considered anathema a decade ago. Not so much the less-than-intuitive notion of test-driven development, but the fact that developers are responsible for low-level testing of *code that they wrote*, and the related fact that such testing must be done in code rather than by banging on the UI. I believe it's critical for developers to engage in these practices so they produce code that is proven to stand on its own two feet (and keep standing on them as time passes). I also believe testers and other customers on a project should hold developers accountable for this testing. Once they are comfortable with these notions, I want developers to know that those tests aren't really just for validation; they should be small, focused specifications of the code's intent. This, to me, is part and parcel of ascertaining the software's suitability to purpose.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
GB: Unfortunately, too many testers today are saddled with energy-sapping, boring repetition of verification tasks. They are supposed to follow test plans established months and years ago that keep growing and expanding. That means spending little thought and having little slack to strategize and reconsider options. In many organizations this lowers testers' stature and the expectations from them. Combined with methods that leave testing to the end of a cycle, when inevitably time must be compressed, this is just not a recipe for respect and reliance on testers' true capabilities.

As more organizations become Agile, the role of tester will change. From post-factum verification, it will move upstream to specification; testers will help drive development by storytesting (specifying high-level tests) just before the associated code is written. They will specify more and validate less, thus engaging more of their system-thinking skills. They will automate tests that deserve automation, so they can continue exploring further scenarios. Rather than just discovering defects late in the game, their work will really become Assurance of Quality. The challenge will be for organizations to accept testers' increased significance and contributions, and for testers to accept the new challenges.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
GB: The question would be: "Given XP's heavy emphasis on test automation, is there still room for testers?"

Yes! Testers on an XP project are considered customers: they help shape the product. That means they partake in writing the stories, composing storytests, and exploratory testing. They constantly analyze and look for gaps and help developers as well as lead customers plug those gaps. There is plenty of brain work for testers.

DDJ: Is there anything else you would like to say?
GB: The average company doesn't employ enough testers.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Keith Braithwaite

Keith Braithwaite is hard core about doing good work. As a Principal Consultant with Zuhlke, he works with teams to help the do the good work they know they are capable of yet do not know how to achieve. One example I particularly like involved building a system to autogenerate trades. While trading systems are complicated in and of themselves, this particular system had to negotiate banking calendars for many different countries in order to determine when and where particular trades would be valid. The development team had mired themselves in a marsh of messiness as they attempted to codify the rules for this system. Keith helped them make their way out of this swamp by having them define examples of the rules rather than the rules themselves. This approach worked well - so well , in fact that no defects have been reported since the system entered production!

Keith was an early adopter of Extreme Programming and Agile and has learned lots about how to succeed (and fail) at adopting these practices for everything from line-of-business applications to embedded software. For the last several years he has been passing along his learnings to the rest of us in the form of blog posts, papers, and presentations, including several talks at the recent Agile 2008.

Here is what Keith has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KB: In my second programming job I was lucky enough to work for a guy called Rob Whittick, from whom I learned a great deal about many things. Rob has a skeleton test framework that he carries around in his head and regenerates at each new place he works (I think the original was in Pascal, we used C++). Anyway, Rob insisted that we programmers write automated tests for every function in every class as we went along and run them at every available opportunity. This did all the good things for us and our code that one might expect. What sticks in my mind, though, is that the tests took on a life of their own and became the way the system was explained to people, up to and including paying customers. This was well before I had heard about any of this "Agile" stuff.

DDJ: How would you describe your testing philosophy?
KB: I'm a very firm believer in test-first and test-driven. Both developers writing unit tests before coding, and users writing acceptance tests (these days "checked examples") before the developers start. I'm developing my thinking on this all the time and my current notion is that passing tests are not best thought of as weak evidence that a system is defect-free (as per Dijkstra's comment) but rather as strong evidence that it does in fact have the specified features. I've come to think of test-first users tests as like the go/no-go gauges that machinists use to quickly check if a piece is in tolerance without actually measuring it.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KB: I'm going to go out on a limb and say: finding defects. We are learning much better ways of not letting defects get in in the first place (or re-discovering how to make old ones tractable), which is much cheaper. Through the influence of the technical practices that come with Agile, testing is turning into something else, something more valuable and more enjoyable.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
KB: Testers have a huge opportunity and a huge challenge at this time. In Agile practice (the way I do it, anyway) we bring testers to the font of the process. We make them the user's friend and aid. Their job is to help the paying customer back the programmers into a corner from which they cannot escape without building the system that the customer needs. And to help the programmers by getting the customers to write down the problems they want solved, rather than their hallucination of the solution they think they'd like.

There is still a role for testers who shake the built system really hard to see what falls off, that must continue. What many teams are finding is that those same skills of asking the awkward questions, finding the edge cases, provoking the unexpected response are at least as valuable as part of requirements engineering than they are in hunting defects. To win these benefits testers need to change their behaviours and overcome many decades worth of bad industry habits surrounding their role. In particular, testers need to push harder for designing in testability, and get involved at the very beginning of any development effort. Testers need to get themselves on board and intimately involved from day 0.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
KB: You should ask me "where do you see testing making the greatest contribution to development projects?"And my answer is "everywhere". Write tests to capture requirements. Count tests written to measure scope. Count tests passing to measure delivery. Use tests to drive design. Use tests to communicate between teams, with customers. Use tests to meet regulatory loads and show compliance without mountains of paper. Oh, and to find defects too. But that last one is just the cherry on the cake.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Michael Corning

Michael Corning is a man on a mission. No small mission, either: his goal is to remake testing at Microsoft, and then throughout the rest of the world. Michael joined Microsoft's Engineering Excellence group in part to pursue this mission, which involves patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative testing approaches from across Microsoft and the industry. Dunno whether he will be successful in this endeavor; regardless, I imagine he and his compatriots are in for a fun ride. (Full Disclosure: I am a founding compatriot.)

If this seems an audacious goal, well, Michael is an audacious guy. He is so passionate about politics - both in general and in specifics - that he is Team Lead for the Microsoft Political Action Committee. As a volunteer planning commissioner for his town he is leading a comprehensive for-the-next-century replanning effort which involves patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative technologies from across Microsoft and the industry. As before, while I don't know whether he will be successful in this endeavor, I imagine he and his compatriots are in for a fun ride.

(Can you guess what he talks about in the testing classes he teaches? Yep: patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative technologies and testing approaches from across Microsoft and the industry.)

Here is what Michael has to say:

]]> DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
MC: The first thing a tester needs to know about testing is that you cannot test what you do not understand. If follows that anything that enhances a tester's understanding of that which they are testing is paramount. When we refer to testable software, an overlooked attribute is understandability. We've made great strides toward working better with our developer colleagues, to the extent the devs are at least writing unit tests (even more progress is possible with devs who adopt a Test Driven Development approach). Other ways testers can enhance their understanding of any tested artifact is to transcribe the model of the artifact they carry in their head to, ideally, a machine-readable format. If you can't model it, you can't understand it; if you can't understand it, you can't test it. Yet another extremely powerful tool for testers to master are design patterns.

DDJ: How would you describe your testing philosophy?
MC: Funny you should ask. I just submitted a paper to the International Symposium on Software Reliability Engineering entitled, "A Tester's Guide to Aristotle's Theory of Virtue - A Model of a Happy Tester." So I guess my answer to you is the same as the answer I gave to Forbes magazine over a decade ago: I'm an Aristotelian tester. Actually, I'm part Neo-Platonic, too. I am convinced that ideas predate awareness, rather like Plato's "Forms." I call them memes. At the moment, my favorite meme is the pattern language. Actually, I'm beginning to rethink my notion of memes: at first, I saw design patterns as memes; now I'm beginning to think memes are patterns, a subset being design patterns. Put a different way: my work on memetic engineering is refocusing as pattern language engineering.

I am a Certified Public Accountant by training. In fact my Master's thesis in 1980 studied whether financial auditors were auditing around or through the computer. This research found that a majority of auditors considered financial accounting software to be a black box. Only a small minority of auditors used white box auditing/testing techniques. This background continues to inform my view of software testing. In my mind, the tester is an auditor. The function of the auditor is to attest. In financial auditing, the auditor attests with his or her signature that the financial results fairly represent the financial results of business operations. From this attestation, other decision makers make their decisions. Both the auditor and the tester provide information to the eco-system. Investors buy or sell based on audited results; product managers ship or hold product based on tested results.

I'm also a scientific tester. This role is closely related to my audit philosophy. I share the view of Philip Armour that the test org is ideally chartered with finding a class of information that neither dev or pm/ux [user experience] colleagues are tuned to. The tester's job is to identify questions (and subsequently answers) about the application under test that have never been considered. We call them UnkUnks (unknown unknowns). Yes, Secretary Rumsfeld took some heat when he tried to explain the role of UnkUnks in the War in Iraq, but political flak notwithstanding, UnkUnks are, by their very nature, un- or underappreciated. The ironic thing about UnkUnks, though, is that they are pervasive, and everyone encounters them or their consequences, and frequently. So common are the UnkUnks that we have a phrase for them: The Law of Unintended Consequences. Virtually everything else I think about testing comes back to the UnkUnks. Design Patterns play a role in understanding the UnkUnks' potential or presence, modeling squeezes them out in the process of model development, and the philosophical approach I take towards the craft of testing also helps find out things about software that designers and developers didn't or couldn't consider.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MC: Though I wouldn't go so far as to denigrate the following as unimportant, I do believe they are generally misunderstood: test case count and code coverage.

You can see from my previous remarks that I spend precious little time thinking about testing in terms of the number of test cases I have. I do my best to use the machine to generate test cases for me. As George Gilder said, "You make money by wasting what's plentiful." Well, my human bandwidth is far from plentiful. My imagination and intuition, my passion for discovery are limitless. This is why I focus so much of my attention on finding ways for testers to use technology to test technology.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
MC: Two things keep me up at night: where is our catalog of the hard problems in test? How can we leverage technology sufficiently to catch up with the complexity of technology?

The catalog of Hard Problems is our collective record of second order ignorance (viz., we have the questions, not the answers). One of our bigger problems in test is that we're the team in charge of coming out of third order ignorance (UnkUnks), and you can't catalog what you don't know you don't know. You can work on describing the process that you can use to identify unknown questions, but the test community hasn't even done a good job with second order ignorance. As a scientific discipline, we lag behind on two fronts: almost all other disciplines exceed ours in second order ignorance and in the use of modeling.

One of the things that helps me sleep at night when I start to worry about the second thing that keeps me up at night is that many of my colleagues, the best and the brightest, are doing work in test at a deliberately abstract level. Modeling and patterns are the two best examples. As our programming languages continue to help us write test code and infrastructure that take advantage of more and more design patterns, we are making progress. As modeling tools become more ubiquitous, we will make even more progress.

One of the most intriguing things I'm watching is the advent of software that enables robotics. In my view, the Microsoft Robotics Studio promises to bring a whole new level of productivity to testing. But I'm a little ahead of the curve here, so additional comments and observations will have to wait for another time....

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MC: Legend has it that as Charles Simonyi (Father of Office Word and Intentional Programming) was leaving Microsoft someone asked him what surprised him most about the company. His answer, "How conservative the Microsoft Developer is." I'd say the same thing applies to testers and to test management. For example, it took almost a decade for model-based testing to take deep root in the test community. There are many reasons for this, but being very conservative is one of them. The other thing that constantly surprises me is just how daunting is the task of testing. I heard that someone analyzed all the dependencies in Windows, alone, and the graph used paper forty feet in length. I've seen how far a test tool like Pex can get into the CLR simply by unit testing a resource provider component, and I'm blown away. Put differently, I've been to no other place where it's possible to feel like you're the smartest guy in the room and the dumbest guy in the room - at the same time. Camus was right: the work is absurdly difficult; and therein we find the nobility of it.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Ethan John

Ethan John professes to not enjoy writing code. Testing, on the other hand, he enjoys immensely. That makes him my kind of Computer Science graduate: the kind who codes only because it gives him an excuse to test. Ethan currently works for Isilon Systems, who I am sure is happy to have the advantage of his love for testing.

Here is what Ethan has to say:

]]> DDJ: What was your first introduction to testing?
EJ: I was in school, and got a job as a research assistant on a project called UrbanSim. It was an Agile house, minus pair programming, so they were doing test driven and iterative development in Java. I had only heard about TDD a few months prior, and my initial experiences with it had been positive. Unit tested code tended to work more consistently out of the gate than otherwise, and I was sold after just a few weeks on the project.

DDJ: What did that leave you thinking about the act and/or concept of testing?
EJ: I had never thought about testing as something unto itself before. Previously in school, testing wasn't something you did consciously so much as it was the necessary evil of debugging something that should otherwise work. Getting down and dirty with JUnit allowed me to see the value in testing as a practice, distinct from just writing code.

It also set me on the path toward testing as a profession. I enjoyed the process of writing unit tests far more than I enjoyed the process of writing the actual code. Thinking through the various kinds of test cases and writing them always left me bored with the process of writing code that actually did the heavy lifting.

It actually took me a while to actually learn that testing was a field unto itself. That seems so obvious now, but the University of Washington teaches a very theoretical CS program, so the various fields available to us as CS majors weren't exactly written down. Even though that first experience with Agile turned me onto testing, I had no idea that it was something I could do for a living.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
EJ: Well, I started out as a developer (or rather, as an aspiring developer), and a lot of the testers that I've worked with don't appreciate the value of system design knowledge. Knowing how a system works as a whole, and having deep technical knowledge about how the code works, is extremely useful for everyone from folks writing white box system acceptance tests to those running through black box Web UI scenarios.

Having a large-system view is important for a lot of reasons. It's useful when debugging ancillary problems that might come up. It's useful for evaluating overall product usability, and for recognizing oddities that developers might dismiss as necessary evils. It's also useful for problems that might arise in ancillary areas. At Isilon, our systems are extremely complex, and early in testing, it's common for item X to malfunction while you're looking at item A, and having a general view of the overall system helps tremendously when investigating those issues.

Some of this system familiarity simply comes with exposure to a product, but there is no substitute for knowing what's going on at a slightly deeper level. Knowing what pieces of the system use which libraries, for example, can be invaluable when trying to figure out an elusive repro, or when trying to write a test plan for a fundamental system design change. Testers in general need to be interested and motivated to learn as much as they can, in as much detail as they can, about the systems that they are working on.

This speaks to something that developers need to be good at -- explaining changes that they make and code that they write to testers. I know it's cliche at this point, but testers and development need to have a very close relationship, and part of that relationship is having a common understanding of the scope of code changes and how they might affect the system as a whole.

DDJ: How would you describe your testing philosophy?
EJ: I view testing as science. Hypothesis, test, conclusion, and repeat. This implies a general view of good testers and how they work. Biologists can't simply pawn off their experiments to people completely inexperienced in the ways of the lab -- there is a baseline of understanding necessary to do anything in a biology lab. Likewise, there is a baseline of knowledge and a scientific mindset necessary to test.

Additionally, testing should be approached as a science. Test cases should be framed in terms of hypothesis/test/conclusion. Sometimes these things are implicit -- if you're testing a Web UI, many of the hypotheses are so common that they needn't be stated. But in the world of system acceptance and other more complicated areas, these hypotheses are often not so well defined, and without them, you often end up testing something that answers no question and provides no data.

Finally, science is constantly informed by history, and testing is no different. Historical results are key to determining how to test in the future. If it provided no benefit to test certain things in the previous cycle, how much benefit is it really likely to provide this time around? Historical data informs future results in the test lab just like in the labs of all other hard sciences.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
EJ: Testing-as-science seems obvious to me. The common test methodologies are rigorous and extremely scientific. I'm surprised by the number of testers that don't see it this way. Many people seem to view testing as something rote (perform these steps, record this result, move on) or as something random ("cowboy" testing, extreme reliance on ad-hoc testing, fuzzing as the whole solution, etc.) rather than as an iterative process that generates a wider view of the stability and usability of a system.

DDJ: What is the most interesting bug you have seen?
EJ: Bug 5112 in our system here at Isilon is about the fans in our nodes being prone to failure upon introduction of "foreign matter." A tester with rather long hair had been standing in such a way that his hair was pulled into the node, causing the rear fans to stop working, and the node to overheat. It was closed WORKSFORME by a developer with no hair at all.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Amol Kher

Amol Kher has been a tester at Microsoft for five years and change. While his feature teams appreciate his contributions to their work, testers across Microsoft appreciate his contributions to theirs: as one of the authors of a popular internal model-based testing tool, Amol is directly responsible for powering up their testing efforts. Amol has been powering up testers in a different way of late with his recent move into people leading, which he is discovering to be every bit as challenging as writing his model-based testing tool has ever been. He has a new challenge at home as well in the person of his infant son.

Here is what Amol has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
AK: When I worked as a developer in a startup with very little test resources. I can't say I was a well-behaved developer, but I thought about the plight of a web application tester and the difficulties in robustness they had using third-party Web UI tools to automate testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AK: It's never enough, also that people seem to think that because there is very little code, that there are not many bugs. Well what about missing code? Also how close testing is to representing your customers and being that first line of defense.

DDJ: What is the most interesting bug you have seen?
AK: I can't say I have rated any particular bug as the most interesting. Bugs that generally triggered QFE's have been quite interesting in general for me but more from the point of view about why did we/I miss it, what did the customer do that I didn't think about.

DDJ: How would you describe your testing philosophy?
AK: Think like the customer does and try out crazy things with your product. Use modeling to better understand and explore interactions between different components under test.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AK: Time is always short, developers always want more features, customers don't always know how to use the features; how do you balance all of that by providing your feedback? Developers should do component level testing and leave the integration and scenario testing to test teams. Our developers should not just write code, they are responsible for writing high quality code, and they are the best people who could avoid bugs by writing unit tests.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
AK: We depend too much on the number of bugs found and lines of code covered as the only metric of quality. I am generally more interested in data coverage and did you verify the method did what it was supposed to do.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AK: Effectively representing your customer when software code is becoming so complex.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
AK: Model-based testing- Why have I seen the light? I just recently found 2 bugs in product code that has been shipped for over 10 years by not even specifically looking for these bugs. Models can do that to you. Models are what developers will hate to love or love to hate.

DDJ: Is there anything else you would like to say?
AK: Enjoy reading your blog!

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Scott Barber

Scott Barber has done a lot of stuff for a lot of people, especially people who test for a living. Especially for people who test performance for a living. Books, directorates, co-founderships, trainings, and the-application-doesn't-work testing - Scott has done it all. Back in the day of Windows NT 4.0, he took the time and trouble and tests to become a Microsoft Certified Systems Engineer. These days, Scott has effectively self-certified himself as a World Renowned Performance Testing Expert via his plethora of books and articles and classes on the topic, all of which arose from his general passion for making performance testing more understandable and more fun for testers everywhere. Maybe even you!

Here is what Scott has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SB: One Sunday evening I got a call from my manager...

"Scott, be at the Marriott Hotel Conference Room at 8am tomorrow. Our CEO has an announcement to make."

Being a good employee, I did as I was told and went to the meeting. As it turns out, our software development company was "merging" with a huge media company. We were going to become their "development branch". As a result, we were giving up our bonuses, overtime, and our base pay was being reduced, but we were going to get stock options.

As one might imagine, I was less than enthusiastic about this. I called my friend of 10 years before I even left the parking lot of the hotel. I told him the circumstances and asked him if he'd help me update my resume, to which he responded:

"Dude, that sucks. Send me your resume I'll take a look, but, we need performance engineers. You'd be a perfect fit."

I said "Performance engineer? What's that?!?"

He replied "Don't worry, you'll like it."

He was right. I've been a tester ever since, and continue to love it.

During my first few projects, I learned just how difficult it is to develop really good software, how challenging it is to find and focus on defects that matter, and how much I enjoyed trying to solve the technical, political, social, schedule, resource, and business problems embedded in all testing efforts. I didn't know it at the time, but I was quite lucky, because as a performance testing consultant I was almost always treated as a peer to the lead architect. In retrospect I realize that was because I was a generalist who understood how systems, not just components, worked from the bottom up as well as from the UI down, who viewed my role as a service provider to the development effort, not as an approval gate on the path toward release.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SB: Over the years I've been shocked to find out that most testers and development groups view testing as a type of QA/QC function. I was shocked to find out that there were testers (as it turns out, lots of them) that were expected to assess "Pass/Fail", to sign-off that a piece of software was "ready to ship", and to generally make business decisions about software and applications, often without even having access to the business reasons for developing the software in the first place. To me, the notion that a tester/test manager should be gating the ship/no-ship decision is baffling. We (testers) are information providers, we are not product managers, account managers, or Vice Presidents of Software Development.

Asking testers to make these "ship/no-ship" decisions, feels to me like asking the CSI's investigating a case to also serve as the judge and jury for that case without so much as a hearing. It seems to me that testers should be providing their results and analysis to the team, the stakeholders and the decision makers so that they have the data they need to make informed technical and business decisions.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SB: I think that the single, most important, thing for a tester to know is where they fit in the team, the project, the company, and what they can do to best serve their stakeholders. I think that the vast majority of testing is done for 1 or more of 4 reasons, but that most testers are not made aware of what their test results are primarily being used for. I think that understanding which one(s) of the following are the primary uses for their test results can dramatically improve the effectiveness of a tester or test team:

To provide information to stakeholders and decision makers so they can make informed business decisions.
To provide information to developers that help them build better applications.
To figure out what problems/challenges/concerns end users will have with the software and communicate that back to the team.
To try to assess compliance of the software or application with legally binding contracts and/or regulations.

I'd like to tell all developers that testers, at least testers in my community, are there to help them make better software.

I'd like to tell all developers that most of the time, these testers are no more interested in defects getting reported to management than they are. That defect reporting is a process "thing", not a tester "thing". That I've met very few testers who aren't happier to skip the bug reporting altogether and just sit with developers and help them "get the bugs out" before the bugs ever get on management's radar.

DDJ: How would you describe your testing philosophy?
SB: As a tester, I am a service provider. The service I provide is quality-related information. The people I serve are developers, stakeholders, end-users, and compliance & regulatory agencies (in that order when given my preference). If I do a good job helping developers build quality software that meets business, end-user, and compliance needs, I've done my job well. When I don't get to work directly with developers, if I can identify issues that threaten business goals of the software early enough to do something about them, I've done my job well. If an application goes into production and the end-users don't experience any issues that I haven't previously identified, I've done my job well. When legal regulations and/or contracts are involved, if I can identify areas that are out of "specification" early enough to get them in "specification", I've done my job well.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
SB: I think there are a huge number of challenges facing both individual testers, and the test discipline over the next 5 (and probably more) years. That said, I think that most of those challenges fall into three broad areas.

Attracting (and Retaining) Good Testers � I recently had the opportunity to meet and work with some software testers in several Latin American countries. Every one of the testers I met are well educated, well trained, well respected members of the team who went to school to become testers, and are proud to be testers. This is simply not the norm. Outside of Latin America, most testers I meet are poorly trained, under respected, of mixed educational background and frequently frustrated with their career. This is a problem. How can we expect to attract folks into testing if so many of our current testers are discontent? I believe that we, testers, need to start viewing ourselves differently if we hope to attract and retain high quality testers. I believe that if we start viewing ourselves as, and acting like, information providers and trusted advisors to software projects, that we will earn greater respect, that our pride in being testers will increase, and that ultimately we will attract and retain increasing numbers of high quality testers.
Educating Testers � Most testers receive all of their training on the job. When they do get external training, it's generally delivered in short 1-3 day courses with no tests, and no homework, that are well known to be a reasonable way to expose someone to new ideas, but lousy at actually teaching anything. The truth is that testing is a wildly diverse discipline that requires the application of knowledge from a wide variety of fields and that most of the best software testers have taught themselves much of what they know that makes them very good testers. We, testers, need to pay more attention to what skills are common across very good testers and then figure out how to teach those skills.
Evaluating Testers � I believe we need to fundamentally change how we evaluate the quality and value of testers, especially during the hiring process. Recruiters, hiring managers, and testers widely agree that today's assessments based on certifications, vendor delivered training, question-based interviews, and laundry lists of technical skills are fundamentally lousy indicators of whether or not someone is a good tester and whether or not they will be a good fit within a particular company or team. We, testers, need to get involved in devising better methods. Mismatches between the hire and the role hurt everyone, and currently, the most effective method we have to get a good fit is the old adage "It takes one to know one."

The Association for Software Testing (AST) is sponsoring projects and programs related to each of these areas, but these kinds of changes take time and lots of work. If you are serious about software testing, I encourage you to join AST and volunteer to assist with the projects or programs that you believe will most help testers individually and collectively face the challenges the future holds for our discipline.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Kyle Larsen

Kyle Larsen water skis barefoot. He also tests Microsoft's security engines. I don't know which feat would scare me more to attempt! Kyle, however, takes them both in stride. If you ever meet Kyle, his love for testing (or for water skiing, depending where you meet him) will come through loud and clear. So will his willingness to help other people learn, whether those other people are young children setting out on their first barefoot ski attempt or young testers setting out on their first bug hunt. Regardless of whether your goal is to increase the quality of your skiing or to increase the quality of your code, Kyle can help you do that.

Here is what Kyle has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KL: My first introduction to formal testing was in a Software Engineering course at South Dakota School of Mines and Technology. What I thought was that it was impossible that, knowing all of the test cases in advance, a development team would not build software that passes every test. The test team gives the answer key to development, so ... that math still doesn't make sense to me.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KL: Model-based testing, monkeys, test generators, and these sorts of things mean there are many easy ways to explore an infinite input space so, it is not as easy for development as it might seem. What surprises me most though is that testing is a bottomless pit. There is more to learn than has been discovered. Even if we could solve every class of defects known today in an instant, it takes only one creative tester to think of a new class of issue that no one before sought to discover and everything comes unraveled.

DDJ: What is the most interesting bug you have seen?
KL: Every bug that is not found on purpose is the most interesting bug. Every bug that is stumbled upon by luck or without any real plan to have found it represents a single sting from a swarm that, left unaddressed, will follow.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
KL: Know the product. Know the developers, their designs, their code, the changes they make, their process, the defects they fear, the mistakes they made in the past, the mistakes the competition made. Know what the users appreciate and what they hate. Know the techniques others are using to discover defects in general and in specific to your own product. Probably what is most important though is to know that there is always something in the product that can be improved.

DDJ: Is there anything else you would like to say?
KL: Are you counting five bullet points or five question marks? One bullet point here was full of three questions and a conjunction to top it off. But, on the other hand, does this count as an answer? I'm sure I lost count somewhere ...

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Jim Bullock

Jim Bullock has been building software for twenty-five years. He has been writing articles and books which engage your mind for almost as long. And I am sure he has been an interesting person for at least as long.

I remember the first time I met Jim in person. I figured we'd talk for an hour or so and then I'd be on my way. Instead, we talked for over four hours. We stopped then only because we each had supper engagements!

One reason Jim and I talked from one meal clear through to the next is that both of us want to understand most everything. Jim has a knack for asking questions which uncover other questions which uncover other questions and so on until he uncovers the root of whatever matter is being discussed. I discovered this helps him be an excellent lunch companion. A plethora of companies have discovered this also helps him move companies from floundering to fabulous.

Jim pays attention to who does what, when. He calls this Conscious Development. I believe he would join me in also calling it fun. Here is what Jim has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JB: I learned to develop software supporting University research labs. Everything was novel, we needed it to work and suspected it wouldn't. So, I was first exposed to software testing with real questions, mostly about my own work, and a personal interest in the answers. That environment was also hugely influential to me - people creating experiments to learn something they wanted to know. We tested our software from the same POV. In testing software I think what you want to know comes first. From that you invent experiments and the gear to support them. We ask software testers to do experiments on software for us because they are crafty and find interesting things.

The first independent software testers I worked with tested my team's heat pump control software at Carrier Corporation. It was both disconcerting and comforting having other people poking at our stuff. Most of the problems found we'd correctly implemented the wrong thing. After that we got testers looking at plans, requirements, designs and so on. You can test anything, and it's worth it. Something happens in people's brains when they think in terms of interrogating the thing in hand. I also learned the value of independent eyes. With those systems we developers were sure what the system should do, and we were wrong. It turns out your own understanding blinds you.

Ever since that experience at Carrier, I try to get ignorant eyes on every work product. To start with, independent testing is a kind of ignorant eyes not blinded by what you thought as you did the work. With code I organize development in a Lean / FDD hybrid with strong gating. Then, every integration gate includes a developer who didn't work on the change and a tester. As new eyes, those reviewers are really a test case, too, standing in for maintainers who come along later. A successful system is going to outlive the original team, so "somebody new" and "recovering history" are two use cases every system will experience. Why not test for them? (Actually the Ingres engineering VP talks about this in the WSJ business technology blog - first week of June, 2008).

I organize testing the same way � Lean / FDD work flow with strong gating and lots of "ignorant eyes" along the way. With testing the extra eyes are especially helpful since testing only gets action when someone else accepts the results. The extra eyes test your delivery as well as your results, and how it plays for someone else, not just for you.

I was first responsible for independent testing while running the implementation of a 3-city, distributed IT system on a new product. I oversaw the vendor's development and testing as part of our product acceptance, and managed our functional testing and specialty tests like performance and data conversion. From this I learned that independent testing also flushes out configuration management & build, requirements management, and development practices � really any kind of communication. Since then, I always make a "process error" defect category for things like these. I also learned that in evaluating your product other people assess how you did the work � explicitly or on the sly. So, it is good to have a compelling, transparent story of what you did and why. Then test your story.

Since then, in PM and development roles (including life cycle tools development, which is fun) I co-created a testing consulting practice in the mid-1990s serving multi-tier enterprise IT development projects. After that I spun up several QA / test teams for both web site companies and products. Each company or project is a little different because different systems have different questions. "Correctness" sure, but correct in what sense? Under what circumstances? By the way, what are the project and company constraints on the testing we can do? Somewhere in there people decided I was a tester. I'm not so sure. I mostly align the team and tools to the needs at hand. Sometimes I get to do testing myself, mostly around architectural risks or performance.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JB: I'm surprised by people focusing on ritual vs. results. I've seen testers insist on doing stuff that finds nothing. I've seen developers insist that "You must test this way" when something else is finding lots of valuable results. I've seen testers and developers ignore impassioned questions about the software because their preferred techniques do something else. I've seen testers & developers ignore the performance that the project or organization needs from testing.

I'm baffled. If testing one way finds nothing, try something else. If testing another way finds things that matter, well, that result is all the justification you need for the technique. If there's something you need to know about the system you have, figure out how to discover it. I think testing starts with questions: "What do we want to do better?" and "What would we like to know more about?" followed by: "How do we find that out?" and "Is the testing we are doing helping?"

I sometimes find testers, developers, PMs & others griping about each other, too. "Those guys take too long." or "Those guys build stuff we can't test." Sometimes people act like those other guys can fail but somehow we'll still be OK. Maybe it's because I bounce between roles but I don't get that. It's not a lifeboat where the guy who falls out is in trouble while we're OK. It's a human chain where if anyone is in trouble we all are. We need these people. Measure your contribution in terms of how you create opportunities for others to contribute in turn.

When I'm PM-guy or running development, I tell testers: "Here's what I could use from you." Then I ask: "What do you need from me to be able to do your job?" and "What else can you offer?" The better they do their job, the easier my job, so why wouldn't I? When I'm doing testing I focus on the kind of information development, sponsors and Software Process Improvement (SPI) need about the system, and the kind of performance in doing testing that will help them succeed. Techniques follow from that.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JB: There's a lot of arguing from position that I don't get - "schools of testing", methods and movements (the current loud one is "Agile"), and individual techniques ("unit tests" and "model driven testing" for two.) I don't get it. The pragmatic people I know go: "Interesting . . ." to anything that does some good, and "Well, duh" to the limitations. Professionals have a kind of informed open-mindedness.

The other day on his blog Keith Braithwaite quoted an old-school paper by D. L. Parnas. He points to an article by Michael Feathers, that refers to something from a while back called "Clean Room Software Development." Clean Room Development is nearly the opposite of "Agile" in many of its choices, and it works, too. (Maybe if you turn the knobs down to 0 that's also interesting.) Testing in Clean Room Development works stochastically from the outside exploring a formal model of the state space the system will encounter. So Feathers and Keith, both of whom would probably call themselves "Agilists", were both looking for interesting ideas, observing what happened, and seeking to understand. Isn't that the testing mindset? Feathers concludes that techniques that work encourage reflecting on how your stuff is supposed to work. That's the right way to do it. Arguing from position is for amateurs.

Arguing from a position in testing is most virulent around techniques. For example "All tests must be automated" is silly, while declining to automate tests you sensibly can is just as dumb. For all the good of the current emphasis on "developer testing" - and by the way, when was developers not testing their work OK? - some folks get narrow-minded about it. Developer testing by itself is sufficient? What about those ignorant eyes? Or more subtly, does xUnit give you a design technique called TDD, an automated regression suite making refactoring safer or a harness for building interesting tests that operate at APIs? I pick "all of the above", and "do what makes sense", but you'll find people arguing from position that it is one or the other.

The thinking seems to get shallow when advocacy sets in. What's left seems like partial descriptions to me. Sensible testing does something like "V model" if you look at it that way, but isn't just that. Clean Room Development's stochastic testing is "black box" but not "black box" in the way many people mean. It's incredibly disciplined and thoughtful. I think a professional tester is able to pick among the different approaches and tools out there. I think a lot of people have a good and useful partial description of testing software. A great example of this is Cem Kaner's paper on 100 different kinds of test coverage. What, 99 of those folks were being silly? Or maybe each thinks their kind of coverage makes sense in their situation. Maybe it does. Now, you pick the kinds of coverage measure that make sense in your situation.

There's a better way:

The risks and priorities of your system and project point to questions you
want to answer better.

The tools & methods that fit come from those questions.

Whatever you discover is valuable information.

Do the most useful thing you can, in the situation at hand.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you?
JB: What doesn't get enough attention and should?

DDJ: What would you answer?
JB: SPI doesn't get enough attention, and the little it gets is wrong. If you're going to have testing or QA (they're different), have ongoing SPI too. Too many teams keep making the same old mistakes then finding them in the same old way. Everybody is busy but it's dysfunctional, literally codependent. Each group creates an artificial need for the other. What would testing do if development eliminated their usual ways of screwing up? What would development do if testing started finding new categories of problems? What is either group doing about the problems that got past the way they work already? If the other guys learn and change how they work, you'll have to change how you work too, and that's uncomfortable. With testing or QA you have data to drive that if you are willing.

I saw this at a small company client where I was Interim QA & Test guy, then Interim Engineering VP. Doing a maintenance release addressing a dozen known, critical problems testing found 60+ more, equally critical defects and bounced every attempted repair at least once. The regression rate of changes was steady over four months. After I moved to Engineering, it was three months of shipping both 50+ repairs / month and new development before testing found their first problem - same guys, same lab vs. the same product. Five months and counting after finding that problem testing had yet to find another. Meanwhile, via investigations to resolve failures in the field engineering identified several new common kinds of faults in the system and set about removing them across the whole product - hundreds, still with no apparent regression.

The difference was data-driven SPI. First, in test we used the defects we discovered to get more crafty about exercising the system. If configuration variations were failing, we'd try more configuration things until that stopped being fruitful. Same with techniques or approaches to testing. Meanwhile, engineering kept working the same way they had. Later, in engineering we had 4 months of data telling us our most common kinds of mistakes - code management screw-ups, problems ill-understood before repair and design-free changes. (That's in DeMarco's "The Deadline" and it fits with my experience. Once you straighten out the life cycle and tool chain at all, the vast majority of "bugs" are under-design.) So, we straightened out code management then added "explain the problem" and "explain how your design works" to change gating and our results got better.

By the way, I don't believe we had no regression over those eight-plus months. I believe we didn't notice any in engineering and didn't find any in testing. That made me nervous. Somebody should have been thinking of other ways to poke at the system, to find other stuff. In test / QA the team and I had identified 10 kinds of testing we could throw at that system - each "kind" a type of fault and a technique to get at them. We'd executed on one, expanding the coverage over 10x. We'd started applying two others when I transitioned to Engineering, and that's where it stopped. Testing wasn't finding much new because they weren't trying anything new, while Engineering was getting smarter about how to work.

For SPI to work we have to get away from thinking that "SPI = progress toward the one true way." I see too much process-ritual in developing & testing software - stuff that Must Be Done(tm). Better to look at what's working and not working for us, here and now. What does our experience tell us? The way we do testing or development is also a test � an experiment in building software a particular way. If what we try doesn't work, try something else. We also need everyone involved in SPI to look first at their performance, then at the performance of development as a whole. Too often "SPI" means "scolding other people." Too often "SPI" means "sub-optimize the whole to optimize my part for me." I've seen testers and developers do that, and it's silly.

In the example I gave we mobilized resources to let people work better. Testing got permission to go after additional questions and ways to poke at the system, as long as they were fruitful. In engineering "strong gating" really gave management cover to engineers taking the time to think through what they were doing. Effective SPI is really about noticing what you could do better, then allocating some resources to doing that. There is no scolding in effective SPI.

DDJ: Is there anything else you would like to say?
JB: Personally, I emphasize the human interactions around testing and the value testing delivers in the organization. I may have a selection bias, but in my experience effective testing and software development depends on interactions first, processes & practices second and specific techniques last. Fix the former and the latter correct themselves.

For example, my "Big Book of Perfect Software Testing" - a lightning talk and one page of notes - talks about the value testing delivers. Twice that I know of it's been included verbatim in other people's training, which is quite cool. My "Calculating The Value of Testing" in STQE (now Better Software) talks about the value of testing as a business function from executives' POV as sponsors.

On human interactions, Brain Branagan and I presented "The Three r's of Software Testing" about ways for testers to keep their heads in the game. Stresses in a project or organization come out in testing, in part because testing done right injects reality. "The Three r's" is about working with that. There's an article that goes with the presentation that I probably should publish. Or look at "Chinese Contracts" on the AYE Conference site. That's about agreements between people working together, like testers, developers and PMs.

Oh, yeah - hire me. I say I do "conscious development." It's amazing what happens when you pay attention to what you are doing when you develop software. If you like what you are hearing contact me - contract or direct / permanent for the right gig in Seattle.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Keith Stobie

Microsoft Test Architect Keith Stobie scales mountains for a living. Currently he is helping Microsoft's Protocol Engineering Team construct model-based tests for Microsoft's wire protocols. A perfect use of model-based testing, you might say; also a project which will take a while to summit. Keith has also worked with the Live Search and Windows Communication Foundation teams, where he undertook similar ventures. He was also an active member of the Web Services Interoperability (WS-I)'s Test Working Group, where he (surprise!) created tools for testing conformance to WS-I profiles. Most all of Keith's work for the last quarter-century, in fact, has involved testing distributed systems.

Keith scales mountains when he's not at work too, albeit ones of a rather more physical nature, like Washington State's Mt. Rainier and Mt. Adams.

Here is what Keith has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KS: It was so long ago, I couldn�t concretely say. In college I spent my last two summers as a programmer intern for GE on their network management software. I frequently encountered bugs in their tools. I was discouraged to learn that we internal users were lowest priority to get a fix - after customer reported bugs and after internal QA/Test reported bugs. Ah to be a tester � my bugs might get fixed sooner.

At the end of the second summer while contemplating my future full time career, my manager suggested I consider Quality Assurance and Test because "you think differently". For one, I expected the product to be able to do whatever the manuals said! (ha ha ha). Most successful developers I know are exceedingly skilled at working around problems to make progress. Reporting the problem is secondary to getting their job done. For me, I run straight into the problem and want it fixed!

Many developers avoid problems by using tried and true (supposedly) examples or existing code - thus avoiding the pitfalls of unexplored areas. I assume I can code from reading the manual. If the manual doesn�t say there is a limit, I presume there is none.

Consequently, when I went interviewing after graduating, I listed on my cover letter I would consider software QA/Testing in addition to software development. This is still relatively rare today. I took an offer from Tandem Computers whose trademark was NonStop computers. They understood quality and appreciated testing - that helped a lot early in my career.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KS: I suppose the lack of any theoretical foundation, especially when I started. Academia has only scratched the surface about providing a sound basis for software testing. Early debates raged around carefully directed testing and random testing (and continue today). We are still learning when random testing (e.g. Fuzz testing) is most advantageous and when structured testing (e.g. boundary value analysis) helps most. Similarly the early work on code coverage metrics relating to block, branch, path coverage and similar criteria was revealing, but not very conclusive. The advent of combinatorics still has some researchers doing fundamentally bad assumptions from a practical testing point of view. On the other hand methods like test prioritization based on code coverage have worked better than I might have expected.

DDJ: How would you describe your testing philosophy?
KS: I like the context driven school of testing. You need to have many approaches in your toolbox from which to choose. No single approach, tool, etc. applies all the time. However, I borrow from all the "schools" of testing as part of my work. I sometimes do rigorous testing (Analytical school) using Model Based Testing to show all state transitions are covered. I�m currently using Factory school for the 300 contractors we have building protocol test suites. I�ve just finished my second batch of reviewer training with emphasis on process and acting as gate keeper (Quality school) for the quality of test suites produced by our contractors.

DDJ: What do you think is the most important thing for a tester to know?
KS: Finding bugs is not the job. Providing information to assess release readiness is. Finding bugs is important. The reason a philosophy of bug finding is good Is because otherwise too often testers drop into a rut of doing merely confirmatory testing - which only partially assesses release readiness. I�ve also seen testers go to the other extreme and only think about tests which will probably find bugs � and ignore whether the product actually basically works.

DDJ: To do?
KS: Besides enjoy their work, they need to be constantly learning and becoming better educated. I am constantly astounded that so many testers that consider themselves professional or career testers have never even read a single book on testing! Most of the best and most senior testers I know love testing, just like I do.

DDJ: For developers to know and do about testing?
KS: I think test first or test driven development is great and more developers need to become "test infected". The most important thing for developers to know about testers is that testers don�t exist to clean up their mess. Good developers know this.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KS: I�ve been lambasting code coverage for quite some time (see my ACM Queue article Too Darn Big to Test). Tools already exist that can automatically get you 100% code coverage. Of course they may not verify a single result, but they execute all of the code. What we need, but have no automated way to measure today, is verification coverage. I�ve required 70% code coverage from developer test suites. Sometimes they get the coverage, but the tests are laughable because there is no verification.

Good test suites have high code coverage, but high code coverage doesn�t necessarily mean a good test suite. Code coverage also says nothing about the equally important aspect of state coverage.

[See my Table Of Contents post for more details about this interview series.]

Five Questions With Edward Miller

Edward Miller isn't the scientist who helped design and develop the Corona reconnaissance satellites. Nor is he the Edward Miller who rode with Jesse James. He is however the founder and CEO of Software Research, Inc., purveyors of web and client/server testing tools. Edward organized the first Florida Software Testing Workshop thirty years ago and has been part of the software testing community ever since. Between publishing articles in IEEE and ACM, organizing the QualityWeek conferences, and building the aforementioned tools, Edward has been supporting testing since before testing was cool.

Here is what Edward has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
EFM: Probably my first introduction to test was when I was working on writing programs to test student programs on an IBM 7094. Well, "trying to write" may be more accurate. That was before I learned what "undecidable" meant!

Later, in a much more serious vein, for a time in the early 1970's I worked in technology that involved attempting to verify quality in terminal guidance for interceptor missiles that actually might have had nuclear release authority for their nuclear warheads intended for anti-ICBM defense. It was such a different era -- when the Soviet Union as a threat was taken very, very seriously -- and missile defense was a serious, serious business.

And, while it was very challenging work it was also kind of frustrating because of the enormous technical difficulty.

One of the most difficult things for me to do as I recall was to explain to high ranking military types -- Generals with two or three or more stars! -- what ACTUALLY was going to be involved in confirming the quality, reliability and integrity of some of that missile-borne software.

By a curious twist, however, the extreme difficulty that we revealed in trying to confirm the software that was doing that kind of guidance and control work led, I'm sure only in small part and in concert with work done by a lot of other teams, to the USA signing on to the ABM treaty. That was a good thing, for sure!

Then later, in the Reagan era, "Star Wars" popped up and again was used to foment a kind of win for the Western Alliance with the demise of the Soviet Union.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
EFM: In a word, how really hard it is to test applications automatically.

For example, back in 2000, we were near the end of the client-server test tool cycle. Our product line, TestWorks, had been a success, but we knew things were changing. So, we looked into the crystal ball to try to see the future and -- no big surprise here, not rocket science -- the future we saw was "the web."

So we said, OK, if you have a web browser enabled application, what is the best way to test it? And the answer was pretty easy, and really obvious: test it from a browser.

That was the easy part, defining the outside concept. Just test the web application from the browser and you're done.

Some seven years later we now have released eValid V8, an IE clone browser that can test ANY kind of web browser application including websites, web services, e-commerce applications, etc. Because the test functionality is all inside the eValid browser the overhead for performing functions is very low, the fidelity of what is done is very high, and the ability to do tests reliably -- even when the web pages change around a lot -- is very good.

But this isn't a product pitch at all. The real point here is that we found out -- in many cases the hard way! -- that it is a lot more complicated to do a good job of automated testing of a web application than you'd think.

Many times I've said to myself, "if we knew then how complicated it was going to be to get this product going we might not have done it." Maybe we benefited from a kind of technical blindness to the underlying complexity of web applications, but that's water over the dam. eValid is a built engine and it functions pretty much as we intended and hoped.

You never really know until you try to build functionality into a web browser to allow you to test those functions just exactly what's actually going on "under the hood"... until you jump under the hood and try to get control of it! That's when the fun begins, and when things that are obvious turn out to be very, very subtle and difficult!

Believe me, it has been a real challenge. As we often say around here, "We don't get bored...".

But it turned out for the best. Our technology now has a U.S. Patent (with more patents applied for), and there is very strong interest in the technical community in the browser-based approach to web testing applications.

DDJ: How would you describe your testing philosophy?
EFM: There's an old line that goes like this; "In testing good news is good news, bad news is good news. The only news in software testing that's bad news is NO news!"

In other words, you have to REALLY pay attention to what you're doing and what you see and what you observe! Everything may turn out to be important, and it takes a very sharp eye and very careful thinking to do testing well. And, if all your tests pass you learn nothing. Worse, if all your tests fail, you learn even less.

Software testing is, in essence, structured experimentation -- a very disciplined kind of experimentation. Certain stimulii produce certain responses in the thing you're testing -- the UUT (the Unit Under Test) -- and the goal is to somehow characterize whether those responses are OK or not, relative to some understanding of what the object is supposed to do.

It's great when there is a "formal specification" you can test against, but most of the time that's unavailable and instead you have to perform your experiments against something of lesser context and detail. And in that case, instinct, experience, good judgement, and -- to be honest -- quite a bit of luck is needed to get a good result.

Not every test has to produce a failure to be valuable. I think too often people focus too much on testing to reveal failures rather than testing to confirm operation. Of course, finding and documenting anomalies is important, but that's maybe only 1% of the whole picture of a "mostly OK application."

Applying these ideas in the eValid technology brought much of the basics of testing into very sharp focus. In the context of testing web browser enabled applications -- which typically are extremely complex and rich with functionality -- even the slightest errors can cause real havoc.

DDJ: What is the most interesting bug you have seen?
EFM: It seems to me you have to parse 'interesting' in two dimensions: intricacy and value. Intricacy as in "how hard to detect", and value as in "how much impact does the defect have?"

Peter Neumann's "Risks" publishes the biggies. Almost always human goofs that have big impact.

But here is a subtle problem we found that might have taken years to find and might have cost users multiple millions.

A hardware product we were working on performs a kind of virtualization of web access -- in the virtual private network space. There is software in the device that scans HTML passages that are coming over the wire and has to manipulate them in a non-invasive way. What we found as the result of applying some very tricky test suites to the device was what looked at first like just a simple LAN-transmission anomaly.

But it was a consistent and repeatable anomaly.

We finally isolated the problem down to a programming error in one of the HTML processing modules, and we created from that guesswork a set of simple HTML passages that were passed through incorrectly every time.

This was very hard to detect -- in fact, it was a problem with the product that had been there for many years. But as the web-page sophistication level increased, this particular defect had increasingly negative effect. We never finalized the total value lost to the client, but it was certainly multiple millions.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EFM: The quick answer: The Web!

The web is in many ways a real mess, yet it has enough redundancy and resiliency so that it thrives anyway. But far too many websites and web applications are vulnerable to all kinds of problems: pages that don't work, sloppy JavaScript that eats performance, new technologies that have (ahem, shall we say) "rough edges". (The guilty will need no accuser here!)

These days it's way beyond "broken links" -- those are there of course -- the issues are at the level of very powerful applications that just don't work reliably enough, or consistently enough. Ask yourself, did you see some website (or web application) in the last week that turned you off? Of course you did; it's unavoidable.

But if I had to list the web quality issues they'd come down to these:

Web 2.0. The implications of the Web 2.0 revolution in collaborative websites is going to create some very tough quality and testing challenges. You have dynamic content, global asynchrony, and a raft of other tough issues to deal with.

AJAX. The hard part about AJAX is that the better it gets the more "Asynchronous" it gets, and doing QA/QC/Testing on something that isn't terribly consistent in when it does what it does is daunting.

Web Development Environments. New systems, like Google's GWT, the Morfik system, and many others like them, make it easy to create big, robust, websites. If you don't do things right, you ALSO wind up with websites that are nearly impossible to do any serious testing on.

Service Level Assurance. How do you successfully assure that a web application, which could be hosted "in the cloud" or anywhere, really is working correctly? This requires Rich Internet Application monitoring, which is a kind of repetitive functional test process.

Competitive Webiste Comparison. As web-based commerce moves closer and closer to being the main mechanism of "doing business" it's inevitable that firms will need to figure out how their website compares with the competition. Right now that's almost a total unknown! The methods we know about are effective, but they are TOO dependent on human interpretation. Like everything in computing it's gotta be automated.

[See my Table Of Contents post for more details about this interview series.]