THE BOOK OF TESTING

Thoughts From a Braidy Tester

January 2008

Mike Zintel's life goal is to be a train driving photographer. While he hasn't achieved that yet, he has achieved many other interesting things. Like being told he couldn't plug his fifth grade science project back in after it caught fire. Like writing a program to manage underground storage tank inventory. Like helping invent Universal Plug And Play.

These days Mike is Director of Development for a not-yet-unannounced product at Microsoft. Here is what Mike has to say:

DDJ: What is the most interesting bug you have seen?
MZ: I’ll answer a different question. I’ll tell you about the most painful bug. I got a contract to write a billing and workflow system for a law office. The platform was a multi-user MPM/II system. About twice a week, the business data would become corrupt. I held disaster at bay for about two months with a patch up utility. I ultimately lost the contract and the hardware company I was working for failed. Sadly, the same week the customer pulled the plug I found and fixed the interleaved update to a file control block. Even more sadly, I wrote the same style of bug into an embedded data collector that queued data through a local hard drive 2 years later. Concurrency is hard.

DDJ: How would you describe your testing philosophy?
MZ: I’ll assume we are not talking about life critical software. I don’t buy that developers can’t test their own code. I do buy that tests, and developers (and frankly customers) train to correct paths and that diversity will uncover new bugs. I think developers should write correct code. And correct tests. And then run them. With automation. Then an independent engineering organization with a quality mandate should add mostly automated diversity to the mix. The combination of automation, tests and dashboard/reporting should be tuned over time to catch more and more bugs. When the code looks settled and usable in practice, a beta should be run. Not so soon that the feedback is noise due to ongoing code churn and not too late that feedback can’t be acted on. Concurrent with the correctness efforts, automation and tests that validate stability and performance under load should be developed and applied. When the product looks solid, it should be shipped, with telemetry to validate real world experience. Telemetry data should be collected and processed with automation. The entire machinery should be continuously improved.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
MZ: Systems are becoming more distributed and the business cycle is shortening. I don’t think my philosophy above works in this environment. Perhaps some kind of automated “sparse testing” is the answer.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

If you are looking to attend a testing conference this year, consider CAST 2008. CAST is not your typical conference. While each session does start with you sitting in rows staring at PowerPoints, each session ends with you discussing anything and everything with the presenter and other attendees, all enabled by a trained facilitator. Sessions are separated by generous buffers of time, so energetic discussions can continue and continue and continue. Imagine a that - a conference where the point is to confer!

If you are looking to present at a testing conference this year, consider CAST 2008. Check out the Call For Papers, then submit your proposal outline by 4 February 2008. That's only a week and a weekend away, so get cracking!

Dunno yet whether I'll be attending. However, Michael Bolton will be. Matthew Heusser told me he is going. Pradeep Soundararajan is flying all the way from India to participate. Jerry Weinberg is keynoting. And that's only the tip of the iceberg!

While CAST will be a lot of adjectives, "boring" isn't one of them!

Posted by The Braidy Tester at 01:00 PM  Permalink |

Eric Jacobson started Life After University as a Teacher Of How To Use Software. That gave him copious opportunities to notice the problems his students had using software and the defects in said software which his students stumbled upon. Eventually he started pointing these problems out to developers and it wasn't long after that until his first official testing gig.

These days Eric works for Turner Broadcasting, where he tests the software that manages which commercials and programming content air when. Hey Eric, schedule me a Star Trek marathon this weekend, will you? <g/>

Here is what Eric has to say:

DDJ: How would you describe your testing philosophy?
EJ: After attending Michael Bolton’s Rapid Software Testing course [RST], I would like to think of myself as a covert RST tester. Covert because not many of my managers dig RST.

My approach is typically as follows but each test session depends on the context.

Select a chunk of AUT [application under test] functionality that devs consider testable, pose questions to business people until I am clear on how they expect the AUT to function, pose questions to devs until I am clear on how they intend the AUT to function. At this point, I usually have some bugs to log before even starting a test. Next, I attack said focused chunk of the AUT. This begins by helping the AUT to do what it feels most comfortable doing. Once it has proven itself to work consistently, I introduce variables (one at a time) that become more complex, until the bugs begin appearing. I let the tests generate from my mind as I witness behavior. I use the popcorn heuristic, and when the bugs begin to stop popping, I end my exploratory session. Finally, I break out my scripted tests, mark them as pass/fail accordingly, and add any valuable ones that were missing. These scripts are fragments, and only contain one step (i.e., do this, expect this). I’m a firm believer in James Bach’s mine field heuristic, and I tend to encourage different paths through the mind field via intentional vagueness.

I spend 20% of my time automating new tests (or fixing broken automated tests). Lately, these tend to focus in areas of performance testing popular paths through the UI and measuring time spans of UI experience and straight service calls. I use a home grown keyword-driven framework wrapped around QuickTest Pro. I’m also fascinated by monkey tests that randomly make decisions on what to do and can run indefinitely, though I only have a few of these and they aren’t good enough to find bugs.

Is that a philosophy?

DDJ: What is the most interesting bug you have seen?
EJ: Unfortunately, I saw this one in production...in an internal HR app I tested. After managers began complaining that performance appraisal comments they had written for their employees began disappearing, I decided to quiz the users and do more testing in said area. I quickly realized the employees could actually delete their manager’s hidden comments by saving new comments of their own. This was a small company and at the time we had no backups to restore. Most managers lost all their free form text for their performance appraisals and didn’t know it until they were about to conduct their face-to-face performance reviews. I was asked to send out an apology email to the entire company for missing that little bug. It was humiliating.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EJ: Though I haven’t read anyone else’s response, I’m sure I’m not alone on this one. Yes, writing manual test cases with details (e.g., UI stuff) is always emphasized and I think it is a major time waster. From my experiences, these test case details are typically ignored upon execution because they are wrong. Which begs the question, should I fix test case details and lose test time or leave them as is, which would leave a trail of incorrect information?

Another typically emphasized metric is the percentage of automated tests. My management loves to set goals like 50% of your test cases should be automated. These metrics are silly because they assume there is value in automating 50% of my tests, which there may or may not be. But I guess if I only write 10 tests, I only have to automate 5 of them. Hmmmm...

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
EJ: You should ask what frustrates me as a tester? I would answer, that there is no stopping point. Developers can sit back and admire a completed app, knowing their code can do what the specs say it should. They only need to provide one solution to a problem. Testers can never sit back and admire a completely tested app because they know if they have more time they can find more bugs. Testers are not providing a solution, we are inventing problems. Yet, as testers, we are constantly answering questions like “Are you done testing?”, “Can we ship?”. These questions are the stuff nightmares are made of.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EJ: As WPF, Silverlight, Flash, and other new UI technologies become more common, there will probably be a period where popular automation tools like QuickTest Pro and SilkTest lag behind in their support. My dev group decided not to use WPF last year, partly because QuickTest Pro did not support it and the folks at Mercury were not aware of any plans to support it.

On the flip side, it appears that Test Driven Development will continue gaining popularity for the next several years. This is good for testers! My dev team currently has over 3500 unit tests, which us testers walk through and use as build acceptance tests. It gives us the piece of mind that a huge chunk of the basic functionality is working. We spend less time chasing the boring environment config problems and basic functionality, and more time looking for the fun, less obvious bugs.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Evan Goldring is a test architect at Microsoft. Today, anyway; before that he was a Group Manager (i.e., he ran an entire product group and managed the people who managed the various disciplines). Before that he was a Dev Manager (which is to say he managed the people who managed the developers for a product group). Before that he was a tester. And before that he was a developer. And before that he was a tester. And before that...you get the picture, I imagine: Evan has switched between development and testing a lot! He says he keeps coming back to testing for the challenge and creativity. Which I guess means development is rote and boring. <g/>

Here is what Evan has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
EG: Like most (all?) developers my first introduction was not because I wanted to test but because my programs had bugs in them. I know I fixed millions of bugs before this, and clearly had to do some amount of testing to find those bugs, but my intro into real formal testing – testing others' code - happened as the result of a bug.

I was visiting with a family friend who was already a professional programmer. I was a Senior in High School and they were helping me wrestle with the question of whether I should major in Computer Science or Computer Engineering. I knew it’d be one of the two, but I really didn’t understand the difference. At some point, he left me to play with some budgeting software he had created for his dad to use. When he came back in the room I told him I found a bug in it. One of the values was coming up negative when it shouldn’t have. He asked if I knew why that was happening and I thought it was pretty obvious and quickly said "Yeah, I’m guessing you are using float instead of a double and I’m overflowing it." What I didn’t realize was that he wasn’t quizzing me, his mind was racing through mathematical and algorithmic errors he could have made but came up empty. It turned out that everywhere else, a float was fine – the values never got that large. But, the total had to be a double.

It was because of that that I landed an internship with his company the next summer where my job was to test desktop software to do mainframe session management. Without being asked, I found automation software to use, devised my own way to measure my "coverage" of the system, and left them with about 80 pages of documentation on the test methodology, instructions on how to run the tests, and a full gap analysis of what I didn’t cover as well as fully documented automation scripts.

They had thought I’d just test the software by using it (ad-hoc testing) and log bugs all summer long. I was under the assumption that clearly this being a professional environment "just" doing ad-hoc testing wouldn’t get me a return job the next summer so I had to step up my game, but I had no idea what that meant so I sort of made it up. Back then, I couldn’t just search the Internet for "software testing". I just thought – if I was in charge what would I have to see from a college intern to be impressed.

I don’t know if that is a statement about how bad they were or how good I was (the company made solid software, made a lot of money, and was bought out by one of the largest software firms for a crazy sum of money before the tech boom so I don’t think they were that bad), but it was a very positive experience for me because I ended up being somewhat self-taught in testing vs. being ramped up by others. By deriving a good test strategy on my own I could better appreciate what others had done in the field by the time I found articles and books on software testing. It’s sort of like learning calculus through memorization vs. derivation.

DDJ: What is the most interesting bug you have seen?
EG: One time I was doing memory leak testing on a streaming media system and found that, under the right conditions we would find an exponentially increasing memory leak happening, but at the same time a similarly decreasing CPU utilization pattern. Since all of us (dev, test, and PM) had inherited this giant mess of code with little documentation, none of us could figure out why we’de see a pattern like this – it just didn’t make sense. Luckily, I had proven myself so the typical route of questioning the tests didn’t happen :).

We set out to figure out what was going on and we found that there was some exponential backoff code that kicked in when the connectivity pattern was in a certain state. That explained the CPU graph, but only further made the memory leak graph unexplainable – you’d expect to see the opposite effect. Well, it turned out that memory wasn’t cleaned up in each case and the algorithm was trying "harder" each time, but waiting longer in between. "Harder" meant doing more work with more connection attempts that weren’t getting cleaned up.

None of this could have been found without collecting a ton of data and graphing them. A picture (created by automated tests), in this case, was worth 1,000 hours of manual testing!

DDJ: How would you describe your testing philosophy?
EG: There are two elements to my philosophy. The first is "Always have a balanced, but wide, test arsenal". If you over invest, you can miss stuff and not be well equipped with the right debugging and diagnostic tools when a certain bug comes around. You have to know when to use automation, know when to use manual testing, and know how to do manual testing efficiently and effectively with "directed ad-hoc" testing. The first thing I do when faced with testing a new component – no matter how big or small – is to create the big picture of what tools I need in my tool box to enable me to attack it from every angle, telling me everything about the system I need. These tools could be anything from software hooks, to well placed “printf’s” (asserts, exception handling, etc), to diagnostic tools, to linear functional regression tests. Once I have the big picture, the second piece kicks in – customer focused testing.

By customer-focused testing, I mean thinking about what sort of bugs each piece, or the depth of each piece, of the picture would find and what functionality it would verify. I prioritize based on the impact that bug would have on the customer. I’ve seen a lot of people, especially junior testers, get excited by finding a corner-case exception but ignore a layout issue as "fit and finish". In reality, the exception could be an extreme corner case – severe, but hardly ever hit – but the layout bug could chop off important text or cause extra mouse clicks in a usage flow that a user goes through 50 times a day. Likewise, looking at how actionable error messages are. Performance (or at least perceived performance) and other characteristics of the software are important to measure. This is how I prioritize the big picture I create and ensure that I build the test tools that will find the most important – from the customers point of view – issues. In other words, I really like to test with the end user in mind.

Once I have the prioritization, I go through and create all the cases as manual ones. That way, at the end of the project, I’ve at least got a record of all the testing I wanted to do and have a backup plan in my back pocket (using an army of manual testers to hit everything). That usually takes on the order of days. Then I start automating, building the tools, etc. Aside from a great backup-plan, this approach has a secondary benefit – you understand the testing you want to do with the automation. The best automation is born from manual processes that were repeated so often that they are well understood and could be replicated easily, without loosing potency, by code. I’m not saying you have to run each test case 100s of times before you can automate, but by at least laying out all the tests as manual ones, it requires you to think through the steps and run through them a few times to make sure they are repeatable and predictable. This small tweak – creating all the tests as manual ones before automating – has saved my teams time and time again either because we can deeply Plan B if some automation doesn’t happen to fit in the schedule or because we thought through the test cases first which saves us from going in the wrong direction and throwing away a lot of work. This is one of the highest return on investment activities you can do on a project.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EG: Testing the parts vs. testing the whole. As I said in my philosophy, you need to find the right balance. I’ve used products, as I’m sure everyone has, that are just dreadful in one way or another. If you talk to individual testers, they will claim their piece is only contributing to that metric in a positive way. However, the sum of the parts is much smaller than the characteristics of the whole. Performance is a great example of this. If you aren’t looking at both the product-wide and the "nuts and bolts" testing of all the pieces that make up the flow of that scenario, you won’t know what the user's experience will be, nor will you know where to look if that scenario’s performance doesn’t meet the bar. I’ve seen many large-scale projects where too much emphasis was put on the parts, but little ownership of the gaps between them existed. This, at least, has been in some of the projects I’ve been exposed to, and this isn’t a comment specific to Microsoft.

This isn’t a direct answer to the question, but I think too much emphasis is often placed on component level testing and I think some of that should be traded for a look at the whole to strike that balance. Using the second part of my philosophy, I’d prioritize against the severity and likelihood of a customer coming across a bug to know what to trade off.

It is admittedly a tricky business. Is a data loss bug that one person out of 10,000 will hit more important to go after than a more benign issue that will cause one out of 1,000 to have to reboot every 10th time they run the program? Is truncated text on a localized build with only 5,000 projected users more important than every users needing 2 more mouse clicks to do a common task more important? At the end of the day, the answer is going to lie in what is motivating and driving the business so the answer could be different in each company, in each product. But, you need to really understand those elements well. Not all bugs are created equal.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EG: We need to learn how to drive quality - in every sense of the word.

Software Engineering is unlike most other Engineering disciplines, in that the notion of "quality" extends far beyond the traditional bounds. Ease of use, doing what customers want and need from their software, easy and unobtrusive upgrades and patches, etc. are all necessary. You find these notions in other engineering disciplines, but few have all of them like software engineering.

I think our challenge is in learning from these other disciplines so it doesn’t take us 100+ years to become as refined as they are in their approach to quality assurance, yet still enable quick turnaround on new features, and driving quality across the entire product development cycle, not as a bolted-on activity that happens near the end. Again, this is not a reflection of Microsoft, but rather what I’ve seen on average across the industry.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Over the holidays I received a review copy of The Practical Guide To Defect Prevention, by Ross Smith, Marc McDonald, and Robert Musson. I have worked with Ross in the past, and I knew he and his team have been doing some interesting work in this area, so I was eager to learn more. I was also looking forward to hearing all sorts of fun stories describing how they have applied these ideas within Microsoft.

Alas, while details were copious stories were sorely lacking. This is my biggest (only, really) complaint about this book. The many ideas it contains sound great, yet without real-life experiences backing them up I am left wondering how many of them have actually been applied, whether they worked or not, and what the authors would do differently next time. Maybe there's an action-packed sequel on the way...

That said, I found this Practical Guide to be chockablock full of interesting techniques for stopping defects before they start. Ideas I found especially interesting include:

• Applying game theory to the problem of incenting people to do tasks they don't really want to do, such as installing a new build of your product every day


• Combining source control churn and function dependency data with various other bits of information to provide a guesstimate as to the riskiness of a specific code change (please tell me this will be part of Visual Studio Team System soon!)


• Using simulation and modeling to optimize your software development process


• Failure Mode, Effects and Criticality Analysis [FMEA] and Fault Tree Analysis [FTA], which appear to me to be more sophisticated (and exotic) forms of the risk-based testing many testers do today

Also covered is the people side of defect prevention. Suggestions for creating a culture of quality, moving quality upstream, and improving communication abound. However, I felt it to be more a series of "you might want to learn about this" pointers than in-depth explanations. Additionally, as with the rest of the book, while some of the ideas (e.g., "Product Information Is Executable") sound wonderful I was unable to determine whether they have actually been implemented or are merely the authors' idea of the way things ought to be. (If the former, please tell me where to buy this magic!)

My favorite part of the book was the three short paragraphs which describe how the Windows Vista team had a small team of elite testers who manually ran through a set of scenario-based test cases for every build. "In the end, the cost of the seemingly 'inefficient' manual testers helped improve the efficiency of the whole system. The return on the small investment was huge. There is a place for manual testing." I am going to be plopping this bit of this book on multiple desks!

I also liked the idea (taken from Dave Olson's Exploiting Chaos) of having one team do prototyping and initial development work and no production work, while another team uses the prototype as a guide in writing the production version. Developers who are skilled at prototyping are often not skilled at polishing, and vice versa; this model leverages the strengths of each. (Although, again, I am left wondering whether this has actually been implemented by anybody, and if so how well it worked.)

Looking at the book as a whole, I think The Practical Guide To Defect Prevention is worth reading if only to get your brain pondering the possibilities. My advice is to flip through the book quickly, casting your eyes over, well, whatever catches your eye. Next, have a slow read through the table of contents, again taking a longer look at topics which stand out to you. Then put the book aside until that point in the future when you are faced with a thorny problem which might be solvable using some of the tools and techniques in this book. That's the time to pull out this Practical Guide and give the relevant portions a thorough going-over. (Their website too: defectprevention.org.)

Posted by The Braidy Tester at 07:30 AM  Permalink |