THE BOOK OF TESTING

Thoughts From a Braidy Tester

July 2007

Jim Moore is a Principal SDET (i.e., experienced tester) at Microsoft who has, in his words, "made a career out of dealing with ambiguity". Jim has been a tester since joining Microsoft ten years ago in the Interactive Media Group where he tested multimedia applications such as Audioman. Next came a stint on the DirectX team, then he helped ship Media Center, and he spent some time working with static and dynamic analysis tools on the Windows Code Quality Team. That set of experience garnered him the position of Test Architect with the Windows Engineering Tools And Release group. Sounds like fun!

A few months ago Jim moved over to the Internet Explorer team where he is focusing on site and application compatibility. In his spare time he writes mobile games and plays "an excessive number of softball games". Here is what Jim has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JM: First job as a Tester was as a contractor for Microsoft writing script for multimedia web controls. At that time (1996), it was all about cranking out volumes of scripts and filing as many bugs as I could. I learned quickly the value of clear repro steps...I also learned that filing the most bugs also meant having to verify all those fixes :)

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JM: Learning that people have PhDs focusing on Testing methodologies or that there’s a whole scientific community working on core testing issues that involve a lot of squiggly line formulas, Theorems, and research papers that read like theses about medical discoveries. I just didn’t realize that testing had its own academic corner of the Computer Science Universe.

DDJ: How would you describe your testing philosophy?
JM: Test is there to measure quality. Test flicks the light switch and counts the number of roaches, Devs are responsible for chasing them down and killing them. Also, If I found it, the customer will find it. In other words, any bug I find (as defined as ‘something unexpected’ or ‘something undefined’) no matter how weird or unlikely, is as valid as any bug that our customers find. But it then becomes an argument about quality vs. schedule vs. feature set, one of those is going to lose.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JM: Testers to know: Know your customer. Try to understand how a defects impact on your customer. If you can’t think of any reason, remember, you are the first customer and you were impacted.

Testers to do: Champion for the customer. Be thorough, creative, and relentless, but also know which battles to fight. Push for adequate time in the schedule for testing.

Devs to know: That testers are paid to assume you didn’t do a perfect job and to pick the nits.

Devs to do: Learn the patterns of your own mistakes and don’t repeat them. While coding, try to anticipate bugs testers will find. Also, challenge the PM on the spec to make sure all ambiguity is clarified and that they haven’t defined any impossible angles.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JM: Bug count. Bug count is not to be ignored, but is often over-relied on as a metric of Test performance. Bug count alone does not define a good or bad tester. But for sure a good tester will enter a high count of quality bugs. Pay attention to bugs your customers are finding, each of those that were not found by Test deserves root cause analysis.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JM: Two things:

• Time: As components get more complex, configuration matrices expand, and rush to market schedules are marched to, testers will continue to struggle with getting enough time allotted for adequate testing.


• Automation vs. Ad-Hoc(usage) testing: There’s a balance between automating tests and allowing time for ‘real-world’ testing, finding that balance will continue to be a struggle. If it were possible to define all possible actions your customers are going to take, it’d then be possible to ‘fully automate’...at least to a degree that protects your customers.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JM: What type of person chooses being a Tester over being a Dev?

Testers are folks who like to tinker, experiment, and have passion for the customer (they are the people who, when confronted with some issue in any given app, say, "WHY ARE THEY DOING THAT!?!?!?!"). They mostly favor breadth vs. depth.

Devs are folks that like to drill deep on a problem, own the architecture, and create the solution. They mostly favor depth vs. breadth.

DDJ: Is there anything else you would like to say?
JM: There is no grand unified theory of code quality. But there is a mostly known set of product requirements and those are best defined by your customers.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Danny R. Faught is an independent consultant in Texas. I always enjoy Danny's frequent Sticky Minds articles, which cover a variety of testing-related topics. If you were at the Conference of the Association for Software Testing a few weeks ago then you heard him in the opening session. And he runs TestingFaqs.org, a compendium of resources for testers. No matter what your area of interest, Danny likely has something for you!

Here is what Danny has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DF: I interviewed with a supercomputer company just before I graduated from college, and they offered me a job as a tester. The money was good, the technology was sexy, and I hadn't heard yet that some people thought testing was a second-class role. The tests were 99% automated, so I got to use my new programming skills. I found out that I'm good at breaking things, and I like being the underdog.

Much of what I learned about testing came not from inside this company, but from other testers I met via the Internet and at conferences. But here are a few lessons learned from that job:

• Don't invent a new programming language just because you're inventing a new test tool.


• You don't have to be a reliability engineer to do load testing, but you do have to be really good at both testing and programming.


• Open source tools are what makes the testing world go around.


• Don't automate tests through a GUI unless you *really* know what you're doing. It's best to watch someone else crash and burn with this than to make the mistakes yourself.


• In most cases you can learn more about testing from resources outside your company than inside.


• QA (quality assurance) work requires different skills than QC (quality control).

DDJ: What is the most interesting bug you have seen?
DF: Oh, there have been so many! There was the Mac application that had its background area turn transparent, letting the desktop peek through its skeleton. There was the auction web site that didn't scrub the html from user input, allowing me to set up a phishing exploit in the test database that surprised people for months after that. There was a handheld device that crashed when it ran out of memory, or else it would kill off a random application to free up memory. It's bugs like these that keep the job interesting.

DDJ: How would you describe your testing philosophy?
DF: The testing approach needs to be tailored for each situation. In most contexts, the best goal for the testing effort is: discovering and reporting information about the quality of a software product in order to support good management decisions. Management is responsible for the quality of the software, and they need to get the information they need to determine whether the software is ready for release.

That pretty well sums it up. I tend to focus on two methods for all the testing I do: exploratory testing and automated testing, with the balance between the two varying widely from one project to the next.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
DF: Scripted step-by-step manual test cases. They're a terrible waste of time, a massively inefficient way to find bugs. Some day someone is going to try session-based exploratory testing or keyword-driven manual testing to prove to the worshipers of scripted testing, including the dreaded government agencies, that we don't have to script our tests in great detail in order to document the fact that we did good testing. If you want to run the tests the same way every time, automate them.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
DF: You should ask whether people should get to know Jerry Weinberg. And I would answer: absolutely! If not the man himself, at least read his books. Jerry taught me that people skills are more important than technical skills in this business. And he taught me that it's okay to be a maverick.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Dave Catlett started at Microsoft in late 1989 as an intern in Product Support. Two years later he joined full time as a tester on the LAN Manager system, where he worked on the French and German versions of LAN Manager Services for Macintosh. That group became part of Windows NT, and so he tested Windows NT Services for Macintosh for the next several years. Next came a test lead position on the WinSock team, which eventually grew to include HTTP, WinHTTP, and WinInet as he helped ship Windows 2000, Windows XP, and Windows Server 2003.

That's a lot of networking experience!

These days Dave is a Principal Test Architect in the Windows Engineering And Tools team, where he researches and implements tools and processes for testability, risk evaluation, and complexity analysis. He sums this up as Q.E.D.: Quality, Engineering efficiency, and Delighting our customers.

Dave is perhaps most well known for his SOCK mnemonic for ensuring and improving testability. Here is what Dave has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DC: Believe it or not, I think it was in the 5th grade. Remember those old TI-30 calculators? Remember how you could enter numbers, like 77345, turn the calculator upside down and its red digits would spell the word “ShELL”? Then you’d get fancy and do something like 71077345 to really impress your friends (does that constitute ad placement?).

One day I sat down and enumerated what I thought had to be every possible combination of numbers that would result in an English word when the calculator was turned upside down (and maybe even a couple of Spanish words). My 5th grade teacher said he was very impressed and asked if he could keep the list. I really think he confiscated the list because the word “bOOZE” appeared on it; this only being 45 years after prohibition had ended. Between that and figuring out how to make a ball-point pen shoot its innards across the room to hit Susan MacDonald in the ear, I’m pretty sure I started my testing career when I was 10.

The ability to generate test cases that cover every possible usage situation or end-user scenario or parameter combination is fundamental to verifying software. As a tester, generating test cases is your bread and butter: how do I break this thing? What conditions has the developer failed to deal with properly? Is this case handled correctly? When I do x, y & z, does it behave as expected?

Therefore, in my view, one of the fundamental pillars of software testing is combinatorial science; something I started in Mr. Smith’s math class under the tutelage of the TI-30’s dull red digits.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
DC: I’ve been most surprised by the math that can be applied to solve testing problems. When I sat in my Discrete Math class in college, trying to understand graph theory while dating my future wife, I didn’t have a clue that it would become an important tool I would use as a software tester in the future. The graph theory, not the dating…at least I think. Anyway, way back then, model-based testing wasn’t even a twinkle in Harry Robinson’s eye, so as I began to learn about state-based testing techniques I was first of all surprised that I actually still had my Discrete Mathematics text book by Norman L. Biggs, and secondly that the graph theory and graph traversal algorithms in the book were very useful in software testing.

DDJ: How would you describe your testing philosophy?
DC: Risk mitigation. That’s really how I’m starting to think of software testing. I no longer think of it in terms of what it is (testing software), but in terms of what the end result of our software testing activities should be; which is really to mitigate risk. Risk of bugs remaining that will negatively impact our customers or our business (which goes back to customers). This has helped me shift from whining about every bug that wasn’t fixed, to being focused on ensuring that the risk the bug represents to our customers and our business has been mitigated to an acceptable level. It has also helped me add a whole new set of “testing” tools to my toolbox that shift more what I would consider quality assurance activities. For example, inspecting specifications to ensure the software is being designed with testability in mind so that we have a clear idea of how to ensure we can validate each and every requirement, use case and end user scenario in an efficient manner.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DC: So much to test, so little time. I think focusing our test efforts on the areas of code that present the highest risk and mitigating that risk with a diverse portfolio of testing techniques is going to be very important to learn. As I have been thinking of testing more in terms of risk evaluation and mitigation, I’ve concluded that sometimes we get stuck on using a specific type of testing methodology and succumb to the adage that if all you have is a hammer, everything looks like a nail. With techniques like model-based testing, we can now generate more test cases than we know what to do with. I think the big challenge is sifting through those test cases and finding the ones that really validate whether the user requirements, use cases and scenarios are working; end to end. That’s why I like John Musa’s Software Reliability Engineering method. It focuses test efforts on areas of code that are most used by customers. Going forward, we need to ensure that we’re not only good at software testing, but we’re really good at proving that as an engineering organization, we’ve all mitigated the risk inherent with the human software development process to a level that delivers a high quality product in a cost effective manner that delights our customers.

Posted by The Braidy Tester at 07:30 AM  Permalink |