THE BOOK OF TESTING

Thoughts From a Braidy Tester

September 2008

Scott Louvau writes software for Microsoft. He doesn't write any software you have ever used, though, unless you have ever worked on his team. The software Scott writes, you see, is the software his team uses to test their product. To paraphrase the commercial, Scott doesn't test the software you use, he makes the testing of the software you use better.

Scott tells me that if he was not in software he would be some sort of craftsman because "I really like the idea of doing a job you can get better at every day". He recounts a Discovery Channel special on how aircraft carriers are built, a process which involves welding together giant panels of steel. The host watched as a welder took a panel which had buckled from the heat of welding it and completely flattened it with a single hit of his hammer. When the host asked the welder how he knew where and how to hit the panel, the welder replied "Twenty-eight years on the job".

Although Scott has only six years on the job so far, I am sure his teammates would agree that he knows where and how to hit their buckled panels in order to flatten them out. Here is what Scott has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SL: Actually, my introduction to testing was an introduction to not testing. I wrote my first database application for my first real job, and excitedly told my boss it was ready to be released. Within two minutes he'd found a bug and within an hour one bad enough that it wasn't usable by customers.

I learned that if you haven't tested it, it doesn't work. If you wonder if something works and try it and it does, the odds are very good that it works because someone earlier wondered the same thing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SL: I guess it's that testing is so easy, and yet so difficult. It's very easy to break something and to think of the basic things about a system that should work and that you should try. It's not so easy to quickly find bad bugs in a relatively good product and explain why they are important. It's incredibly hard to choose the right tests from all of the possibilities and get them running more cheaply than the code they test.

DDJ: What is the most interesting bug you have seen?
SL: There are so many possible categories.

The worst bug I found caused Visual Studio to crash and delete as much of itself as it could.

The most entertaining bug I found had the offending developer's name burned into it. His My Documents folder became the default for all projects for anyone.

The subtlest bug I found was one where the arrow keys didn't work in one particular combobox in one dialog for some still unknown reason.

I can't sit down and look for bugs for more than half an hour without smiling. It's amazing how many little things there are to go wrong.

DDJ: How would you describe your testing philosophy?
SL: I guess I would say "to be pragmatic". Ultimately my goal is to get the most useful possible product out. I start with the things my product must be able to do and work my way into the soft spots I find as I go. I try to blend automating to prove the product works every day with exploring to find the areas that need more polish to really be useful to people. It's an interesting balance.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SL: I guess it's to remember that we all have the same overall goal - to produce the right product for our customers. I found the process of triage (deciding which bugs to fix and which not to) very frustrating until I really internalized the idea that we're building the same thing.

Remember that many users will have to use your product when you release it, and will cope with the bugs you've created for them every day. Remember that they are also dealing with not having your product every day it isn't there.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
SL: Metrics. They can be useful, but are so often abused to try to force things to be "done". Ultimately it's up to the people working on a project to really decide when it's done, and the metrics should only be potential tools to guide their efforts. If you would not feel good signing your name on the box of the product you've built, don't release it yet.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
SL: I think our hardest challenge will be to cope with the multi-core/many-core transition that's happening. Across computing I don't think we're great at writing massively multi-threaded code, testing it effectively, and reproducing and fixing defects. We will need to be.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
SL: Three questions:
What do you consider the key(s) to your success?
I think two things have served me very well. First, I don't like to just use something that works. I want to know how it works and why it works. Second, I try to do something better every day - automate something I've done manually, figure out a little code design problem - something. It adds up. =)

What is the most important thing you've learned as a tester?
The same thing I said for "What do you think is the most important thing for a tester to know" - we're all working toward the same goal.

Why do you like being a tester?
I think that software testing is much more of a "frontier" than development. We collectively know much more about how to write services and applications and utilities than software tests. I think testing is also much more open ended - anything you can think of to decide whether the code is good is fair game. I guess I'm also just fascinated to see my programs using other programs like a user would.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Michael Kelly is a tester's tester. He talks about testing on his blog as well as in magazines ranging from InformIT to Better Software. He talks about testing at the Indianapolis Workshops on Software Testing he co-founded as well as at conferences ranging from the Indianapolis Quality Assurance Association Quality Enrichment Conference to the upcoming Pacific Northwest Software Quality Conference. And he tests software for companies large and small as well as for the fun of it.

Regardless of how you come to know Michael, his passion for improving the craft of testing and for helping testers improve themselves comes shining through. Here is what Michael has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MDK: I started my career as an intern doing GUI-level test automation in a Fortune 500 company. In a matter of months I had found some of the works on that topic by Kaner, Pettichord, Marick, and others and was applying everything I could to the problems we were working to solve. After less than a year, myself and other $12-an-hour interns were teaching the $150-an-hour consultants how to write more maintainable tests and how they could get better coverage by making simple changes. The experience left me with an immediate sense of skepticism on the industry. I wondered at how those well-paid, well-meaning, and otherwise intelligent people could do work so poorly.

Once I entered the workforce proper, it didn't take me long to figure out testing would be my permanent home. It also didn't take me long to figure out what the problem had been. I found that most popular testing literature trivialized the work that we do and that people actually believed what tool vendors said. It was amazing to me that a group of twenty-year-olds could see through the marketing hype and experienced professionals couldn't. So, I sought out the only group of people I saw questioning the status quo. It was at the Austin Workshop on Test Automation that I eventually found the community that I could call home. They questioned and challenged everything, asked for empirical evidence of claims, and viewed testing as a cognitively-complex and interdisciplinary profession.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MDK: What has surprised me the most is how much I keep learning. I still feel like I did as an intern. There's too much to read, too much to try, not enough time to practice, and never enough time to collaborate with others. I know that I know more then I did ten years ago, but when I look at everything I still only have a passing understanding of, I feel overwhelmed! It's not simply a problem of keeping up with technology, it's also finding time to learn our craft's history and to explore what I can learn about testing from philosophy, art, mathematics, and business; a few of the places from where I've developed my understanding of what we do as testers.

DDJ: How would you describe your testing philosophy?
MDK: I identify myself as a member of the context-driven community, and find that much of my testing philosophy mirrors the work done by James Bach, Jon Bach, Michael Bolton, Jonathan Kohl, Cem Kaner, and others doing work in rapid testing and exploratory testing. I think most testers would be well served by developing a deeper understanding of scientific and systems thinking, heuristics and biases, and project and business management. I would also encourage them to explore areas of interest outside of software testing and to bring something back to the field from those interests. As I write this, we just finished the third conference for the Association for Software Testing and at the conference we had talks pulling lessons from music, magic, lacrosse, finance, improv comedy, statistics, and civil engineering.

DDJ: What is the most interesting bug you have seen?
MDK: I've written about this before, but I remember my first security bug. It was so simple, I stumbled over it accidentally. (Well, I told the very angry people who were upset with me that it was an accident.) The problem started with a developer who had left his or her user ID in a code comment on the login page for a production system. It looked something like this:


<TABLE cellSpacing = 0 cellPadding = 1 border = 0 width = "98%" align = "center">
 <TBODY>
 <tr>
  <td class = "wb">
  <table border = 0>
   <FORM method = "POST" name = "login" action = "/login" enctype = "application/x-www-form-urlencoded">
    <tr>
    <td class = sb>
     login
     <input type = "text" name = "login_id" size = "32" maxlength = "64"/>
    </td>
    <td class = small>
     enter your username
    </td>
    </tr>
    <tr>
    <td class = sb>
     password
     <input type = "password" name = "password" size = "16" maxlength = "32"/>
    </td>
    <td class = small>
     enter your password
    </td>
    </tr>
    <tr>
    <td class = sb align = right>
     <input type = "submit" name = "submit" value = "login"/>
     <!-- u2x34t - Oct 12, 2004: Removed link to defect tracking system-->
    </td>
    </tr>
   </form>
  </table>
  </td>
 </tr>
 </TBODY>
</TABLE>


Note the comment enclosed in the <!-- --> tags. Out of curiosity, I entered the user ID (u2x34t) from the source code into the username field and tried to guess the password. I was rewarded with this:

"The password you entered is incorrect. Please try again."

I say rewarded because the first user ID I tried gave me this error:

"The user id you entered is not recognized. Please try again."

The specificity of the error messages for the system indicated that I was on the right track. I wouldn't have known that if the system had consistently displayed an error message similar to this one:

"The user id and/or password you entered are incorrect, please try again."

At this point, I knew I had a valid user ID, but I still didn't have the password. I didn't want to simply guess because I didn't want to lock the ID (earlier tests had shown that to be a problem). Instead, I started to wonder about that comment the developer made in the source code:

"Removed link to defect tracking system"

I asked myself some questions: What tracking system? How did they remove it? And where did it go? I looked at the source for more clues, but none could be found, at least to my untrained eye. I needed more source code. I figured that if the developer left comments on the login page, there was a good chance he or she left them in other code as well. At the bottom of the login page was a link to a help page. ("Need help logging in? Forgot your ID?") I followed that link and looked at the source for the help screen, where I found something similar to this:


<TABLE cellSpacing = 0 cellPadding = 1 border = 0 width = "98%" align = "center">
 <TBODY>
 <tr>
  <td class = "wb">
  <table border = 0>
   <tr>
   <td class = sb align = right>
    <!--<a href="http://URLForDefectTrackingSystem/%userIdParameter%.aspx">
    Submit a ticket.</a>-->
   </td>
   </tr>
   <tr>
   <!--Help text was here...-->
   </tr>
  </table>
  </td>
 </tr>
 </TBODY>
</TABLE>

To remove the link to the defect tracking system, the developer had simply commented out the link. Not only that—the link included a parameter for the user currently logged in so it would know who submitted the ticket. At that point, I had a URL that required a user ID, and I had a user ID. I simply copied the URL, pasted it into the address bar, typed the user ID in the appropriate place, and hit Enter. The system displayed an error page for a defect tracking system, stating that the system was no longer in use. I initially thought I had hit a dead end, but then I saw a link at the bottom of the page: "Return to application." I clicked the link and was rewarded with the home page (in production) for the application I was attempting to access. No password required!

After that, I was hooked, and I had to learn more about security testing.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MDK: I think there is a large focus on counting things in our industry. Many people count tests, use cases, defects, and even testers. We're obsessed with counting. I hear people say things like, "We have 4,000 tests" or "We found 50 defects" without any indication about what's actually being covered by the tests and what risks are addressed or what severity the defects are. There seems to be an impression that everything is equivalent or that if you just communicate a number you've communicated some meaning.

I don't find counting to be useful outside of its use as a general indicator for size or volume within a specific project context. On past projects, I've seen people count tests like they are lines of code. I find that you fall into all the same problems programmers run into when they do that. It may be a useful metric in some instances, but rarely is it a meaningful metric.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Dmitri Klementiev tests software at Microsoft. No, he develops software for Microsoft. No, he tests software for Microsoft. I'm confused.

Oh, I get it: Dmitri is the type of tester for whom Microsoft originally coined the title Software Development Engineer in Test: he spends his days developing software to help other testers better do their job. In Dmitri's case this has largely meant putting massive amounts of time into developing the tool many Microsoft testers use to automate their drive-my-application's-user-interface test cases. Dmitri's Doctorate Of Mathematics and experience in mathematical and computer modeling seems to be holding him in good stead as he navigates the intricacies of automating the user interfaces of Microsoft's many different applications and operating systems.

Here is what Dmitri has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DK: When I joined Microsoft in June 1998. Automation that can be greatly improved and increase the efficiency of testing.

DDJ: How would you describe your testing philosophy?
DK: We have to deliver high quality products and our customers should love our products which means an elegant and "comfortable" (for users, not for us) design. Testing is a "quality gate keeping" in this process.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DK: Testers - their product, technologies that are used in the products, customer expectations, industry standards, other related products on the market. Programming skills to write automation. Should drive the quality of the products we ship.

Devs - how the products are tested: test plans, approaches, automation. Also know and support requirements to make the tester’s job (including automation) more efficient.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DK: Keep up with new technologies. Make automation much more efficient.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
DK: What’s the difference between a porcupine and a hedgehog. And I would answer "I don’t know" :)

DDJ: Is there anything else you would like to say?
DK: Since I am in UI automation world I would say that we are still pretty much in the beginning of the game - a lot of very complex and at the same time very interesting problems ahead. Let's be those who find good solutions for them.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Gil Broza broke Edgar Allen Poe's second cipher. Gil Broza is a certified NLP Master Practitioner. Gil Broza is living what he describes as a "wild life" as he learns how to raise twins.

Perhaps more germane to this blog, however, is that Gil Broza has been working on software for close to two decades and has been XPing and Agileing for much of that time. Regardless of whether you have a financial, bioinformatics, or content delivery system to maintain or to build, Gil can show you, coach you, and help you do so in an Agile and Industrial XP fashion. You may have heard Gil talking at Agile 2008, at DDJ's own SD Best Practices conference, or at one of Industrial Logic's workshops on various XP-related topics.

Here is what Gil has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
GB: While many organizations prefer to hire intelligent, bright, hard-working testers, some companies appear to think that any warm body will do. Even after almost two decades in the software world, this mindset still amazes me. I guess it arises from the concept of the test plan, or from the notion that given a feature's spec and a working build, a person should really just need to "bang on it" for X hours to prove its correctness. As a firm believer in Exploratory Testing and think-once-then-automate, I think testers require careful hiring just like any other role.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
GB: A critical principle in software development, I believe, is the concept that "testing" and "tests" aren't about finding bugs. Rather, they ascertain the software's fitness to the customer's purpose and goals.

As an XP trainer and coach, I teach developers many practices that were considered anathema a decade ago. Not so much the less-than-intuitive notion of test-driven development, but the fact that developers are responsible for low-level testing of *code that they wrote*, and the related fact that such testing must be done in code rather than by banging on the UI. I believe it's critical for developers to engage in these practices so they produce code that is proven to stand on its own two feet (and keep standing on them as time passes). I also believe testers and other customers on a project should hold developers accountable for this testing. Once they are comfortable with these notions, I want developers to know that those tests aren't really just for validation; they should be small, focused specifications of the code's intent. This, to me, is part and parcel of ascertaining the software's suitability to purpose.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
GB: Unfortunately, too many testers today are saddled with energy-sapping, boring repetition of verification tasks. They are supposed to follow test plans established months and years ago that keep growing and expanding. That means spending little thought and having little slack to strategize and reconsider options. In many organizations this lowers testers' stature and the expectations from them. Combined with methods that leave testing to the end of a cycle, when inevitably time must be compressed, this is just not a recipe for respect and reliance on testers' true capabilities.

As more organizations become Agile, the role of tester will change. From post-factum verification, it will move upstream to specification; testers will help drive development by storytesting (specifying high-level tests) just before the associated code is written. They will specify more and validate less, thus engaging more of their system-thinking skills. They will automate tests that deserve automation, so they can continue exploring further scenarios. Rather than just discovering defects late in the game, their work will really become Assurance of Quality. The challenge will be for organizations to accept testers' increased significance and contributions, and for testers to accept the new challenges.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
GB: The question would be: "Given XP's heavy emphasis on test automation, is there still room for testers?"

Yes! Testers on an XP project are considered customers: they help shape the product. That means they partake in writing the stories, composing storytests, and exploratory testing. They constantly analyze and look for gaps and help developers as well as lead customers plug those gaps. There is plenty of brain work for testers.

DDJ: Is there anything else you would like to say?
GB: The average company doesn't employ enough testers.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Keith Braithwaite is hard core about doing good work. As a Principal Consultant with Zuhlke, he works with teams to help the do the good work they know they are capable of yet do not know how to achieve. One example I particularly like involved building a system to autogenerate trades. While trading systems are complicated in and of themselves, this particular system had to negotiate banking calendars for many different countries in order to determine when and where particular trades would be valid. The development team had mired themselves in a marsh of messiness as they attempted to codify the rules for this system. Keith helped them make their way out of this swamp by having them define examples of the rules rather than the rules themselves. This approach worked well - so well , in fact that no defects have been reported since the system entered production!

Keith was an early adopter of Extreme Programming and Agile and has learned lots about how to succeed (and fail) at adopting these practices for everything from line-of-business applications to embedded software. For the last several years he has been passing along his learnings to the rest of us in the form of blog posts, papers, and presentations, including several talks at the recent Agile 2008.

Here is what Keith has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KB: In my second programming job I was lucky enough to work for a guy called Rob Whittick, from whom I learned a great deal about many things. Rob has a skeleton test framework that he carries around in his head and regenerates at each new place he works (I think the original was in Pascal, we used C++). Anyway, Rob insisted that we programmers write automated tests for every function in every class as we went along and run them at every available opportunity. This did all the good things for us and our code that one might expect. What sticks in my mind, though, is that the tests took on a life of their own and became the way the system was explained to people, up to and including paying customers. This was well before I had heard about any of this "Agile" stuff.

DDJ: How would you describe your testing philosophy?
KB: I'm a very firm believer in test-first and test-driven. Both developers writing unit tests before coding, and users writing acceptance tests (these days "checked examples") before the developers start. I'm developing my thinking on this all the time and my current notion is that passing tests are not best thought of as weak evidence that a system is defect-free (as per Dijkstra's comment) but rather as strong evidence that it does in fact have the specified features. I've come to think of test-first users tests as like the go/no-go gauges that machinists use to quickly check if a piece is in tolerance without actually measuring it.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KB: I'm going to go out on a limb and say: finding defects. We are learning much better ways of not letting defects get in in the first place (or re-discovering how to make old ones tractable), which is much cheaper. Through the influence of the technical practices that come with Agile, testing is turning into something else, something more valuable and more enjoyable.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
KB: Testers have a huge opportunity and a huge challenge at this time. In Agile practice (the way I do it, anyway) we bring testers to the font of the process. We make them the user's friend and aid. Their job is to help the paying customer back the programmers into a corner from which they cannot escape without building the system that the customer needs. And to help the programmers by getting the customers to write down the problems they want solved, rather than their hallucination of the solution they think they'd like.

There is still a role for testers who shake the built system really hard to see what falls off, that must continue. What many teams are finding is that those same skills of asking the awkward questions, finding the edge cases, provoking the unexpected response are at least as valuable as part of requirements engineering than they are in hunting defects. To win these benefits testers need to change their behaviours and overcome many decades worth of bad industry habits surrounding their role. In particular, testers need to push harder for designing in testability, and get involved at the very beginning of any development effort. Testers need to get themselves on board and intimately involved from day 0.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
KB: You should ask me "where do you see testing making the greatest contribution to development projects?"And my answer is "everywhere". Write tests to capture requirements. Count tests written to measure scope. Count tests passing to measure delivery. Use tests to drive design. Use tests to communicate between teams, with customers. Use tests to meet regulatory loads and show compliance without mountains of paper. Oh, and to find defects too. But that last one is just the cherry on the cake.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |