Five Questions With Keith Stobie

Microsoft Test Architect Keith Stobie scales mountains for a living. Currently he is helping Microsoft's Protocol Engineering Team construct model-based tests for Microsoft's wire protocols. A perfect use of model-based testing, you might say; also a project which will take a while to summit. Keith has also worked with the Live Search and Windows Communication Foundation teams, where he undertook similar ventures. He was also an active member of the Web Services Interoperability (WS-I)'s Test Working Group, where he (surprise!) created tools for testing conformance to WS-I profiles. Most all of Keith's work for the last quarter-century, in fact, has involved testing distributed systems.

Keith scales mountains when he's not at work too, albeit ones of a rather more physical nature, like Washington State's Mt. Rainier and Mt. Adams.

Here is what Keith has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KS: It was so long ago, I couldn’t concretely say. In college I spent my last two summers as a programmer intern for GE on their network management software. I frequently encountered bugs in their tools. I was discouraged to learn that we internal users were lowest priority to get a fix - after customer reported bugs and after internal QA/Test reported bugs. Ah to be a tester – my bugs might get fixed sooner.

At the end of the second summer while contemplating my future full time career, my manager suggested I consider Quality Assurance and Test because "you think differently". For one, I expected the product to be able to do whatever the manuals said! (ha ha ha). Most successful developers I know are exceedingly skilled at working around problems to make progress. Reporting the problem is secondary to getting their job done. For me, I run straight into the problem and want it fixed!

Many developers avoid problems by using tried and true (supposedly) examples or existing code - thus avoiding the pitfalls of unexplored areas. I assume I can code from reading the manual. If the manual doesn’t say there is a limit, I presume there is none.

Consequently, when I went interviewing after graduating, I listed on my cover letter I would consider software QA/Testing in addition to software development. This is still relatively rare today. I took an offer from Tandem Computers whose trademark was NonStop computers. They understood quality and appreciated testing - that helped a lot early in my career.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KS: I suppose the lack of any theoretical foundation, especially when I started. Academia has only scratched the surface about providing a sound basis for software testing. Early debates raged around carefully directed testing and random testing (and continue today). We are still learning when random testing (e.g. Fuzz testing) is most advantageous and when structured testing (e.g. boundary value analysis) helps most. Similarly the early work on code coverage metrics relating to block, branch, path coverage and similar criteria was revealing, but not very conclusive. The advent of combinatorics still has some researchers doing fundamentally bad assumptions from a practical testing point of view. On the other hand methods like test prioritization based on code coverage have worked better than I might have expected.

DDJ: How would you describe your testing philosophy?
KS: I like the context driven school of testing. You need to have many approaches in your toolbox from which to choose. No single approach, tool, etc. applies all the time. However, I borrow from all the "schools" of testing as part of my work. I sometimes do rigorous testing (Analytical school) using Model Based Testing to show all state transitions are covered. I’m currently using Factory school for the 300 contractors we have building protocol test suites. I’ve just finished my second batch of reviewer training with emphasis on process and acting as gate keeper (Quality school) for the quality of test suites produced by our contractors.

DDJ: What do you think is the most important thing for a tester to know?
KS: Finding bugs is not the job. Providing information to assess release readiness is. Finding bugs is important. The reason a philosophy of bug finding is good Is because otherwise too often testers drop into a rut of doing merely confirmatory testing - which only partially assesses release readiness. I’ve also seen testers go to the other extreme and only think about tests which will probably find bugs – and ignore whether the product actually basically works.

DDJ: To do?
KS: Besides enjoy their work, they need to be constantly learning and becoming better educated. I am constantly astounded that so many testers that consider themselves professional or career testers have never even read a single book on testing! Most of the best and most senior testers I know love testing, just like I do.

DDJ: For developers to know and do about testing?
KS: I think test first or test driven development is great and more developers need to become "test infected". The most important thing for developers to know about testers is that testers don’t exist to clean up their mess. Good developers know this.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KS: I’ve been lambasting code coverage for quite some time (see my ACM Queue article Too Darn Big to Test). Tools already exist that can automatically get you 100% code coverage. Of course they may not verify a single result, but they execute all of the code. What we need, but have no automated way to measure today, is verification coverage. I’ve required 70% code coverage from developer test suites. Sometimes they get the coverage, but the tests are laughable because there is no verification.

Good test suites have high code coverage, but high code coverage doesn’t necessarily mean a good test suite. Code coverage also says nothing about the equally important aspect of state coverage.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink