Five Questions With Dave Catlett

Dave Catlett started at Microsoft in late 1989 as an intern in Product Support. Two years later he joined full time as a tester on the LAN Manager system, where he worked on the French and German versions of LAN Manager Services for Macintosh. That group became part of Windows NT, and so he tested Windows NT Services for Macintosh for the next several years. Next came a test lead position on the WinSock team, which eventually grew to include HTTP, WinHTTP, and WinInet as he helped ship Windows 2000, Windows XP, and Windows Server 2003.

That's a lot of networking experience!

These days Dave is a Principal Test Architect in the Windows Engineering And Tools team, where he researches and implements tools and processes for testability, risk evaluation, and complexity analysis. He sums this up as Q.E.D.: Quality, Engineering efficiency, and Delighting our customers.

Dave is perhaps most well known for his SOCK mnemonic for ensuring and improving testability. Here is what Dave has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DC: Believe it or not, I think it was in the 5th grade. Remember those old TI-30 calculators? Remember how you could enter numbers, like 77345, turn the calculator upside down and its red digits would spell the word “ShELL”? Then you’d get fancy and do something like 71077345 to really impress your friends (does that constitute ad placement?).

One day I sat down and enumerated what I thought had to be every possible combination of numbers that would result in an English word when the calculator was turned upside down (and maybe even a couple of Spanish words). My 5th grade teacher said he was very impressed and asked if he could keep the list. I really think he confiscated the list because the word “bOOZE” appeared on it; this only being 45 years after prohibition had ended. Between that and figuring out how to make a ball-point pen shoot its innards across the room to hit Susan MacDonald in the ear, I’m pretty sure I started my testing career when I was 10.

The ability to generate test cases that cover every possible usage situation or end-user scenario or parameter combination is fundamental to verifying software. As a tester, generating test cases is your bread and butter: how do I break this thing? What conditions has the developer failed to deal with properly? Is this case handled correctly? When I do x, y & z, does it behave as expected?

Therefore, in my view, one of the fundamental pillars of software testing is combinatorial science; something I started in Mr. Smith’s math class under the tutelage of the TI-30’s dull red digits.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
DC: I’ve been most surprised by the math that can be applied to solve testing problems. When I sat in my Discrete Math class in college, trying to understand graph theory while dating my future wife, I didn’t have a clue that it would become an important tool I would use as a software tester in the future. The graph theory, not the dating…at least I think. Anyway, way back then, model-based testing wasn’t even a twinkle in Harry Robinson’s eye, so as I began to learn about state-based testing techniques I was first of all surprised that I actually still had my Discrete Mathematics text book by Norman L. Biggs, and secondly that the graph theory and graph traversal algorithms in the book were very useful in software testing.

DDJ: How would you describe your testing philosophy?
DC: Risk mitigation. That’s really how I’m starting to think of software testing. I no longer think of it in terms of what it is (testing software), but in terms of what the end result of our software testing activities should be; which is really to mitigate risk. Risk of bugs remaining that will negatively impact our customers or our business (which goes back to customers). This has helped me shift from whining about every bug that wasn’t fixed, to being focused on ensuring that the risk the bug represents to our customers and our business has been mitigated to an acceptable level. It has also helped me add a whole new set of “testing” tools to my toolbox that shift more what I would consider quality assurance activities. For example, inspecting specifications to ensure the software is being designed with testability in mind so that we have a clear idea of how to ensure we can validate each and every requirement, use case and end user scenario in an efficient manner.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DC: So much to test, so little time. I think focusing our test efforts on the areas of code that present the highest risk and mitigating that risk with a diverse portfolio of testing techniques is going to be very important to learn. As I have been thinking of testing more in terms of risk evaluation and mitigation, I’ve concluded that sometimes we get stuck on using a specific type of testing methodology and succumb to the adage that if all you have is a hammer, everything looks like a nail. With techniques like model-based testing, we can now generate more test cases than we know what to do with. I think the big challenge is sifting through those test cases and finding the ones that really validate whether the user requirements, use cases and scenarios are working; end to end. That’s why I like John Musa’s Software Reliability Engineering method. It focuses test efforts on areas of code that are most used by customers. Going forward, we need to ensure that we’re not only good at software testing, but we’re really good at proving that as an engineering organization, we’ve all mitigated the risk inherent with the human software development process to a level that delivers a high quality product in a cost effective manner that delights our customers.

Posted by The Braidy Tester at 07:30 AM  Permalink