Five Questions With Ross Smith

Ross Smith has held a variety of jobs in his life: cartooning, guarding jails, managing unions, and that voice on the other end of the line when you call Microsoft Product Support. Then he found testing and hasn't looked back since. As a Tester, Test Lead, Test Manager, Test Architect, and now Director of the Windows Core Security Test Team, he has been involved with Windows and Office for twelve years running. His recently published The Practical Guide to Defect Prevention is one way he aims to share all that experience with the rest of us. (See my review elsewhere on DDJ.)

Here is what Ross has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
RS: It’s surprising to me how often people make mistakes. Human error is a fascinating topic - from nuclear disasters and bridge collapses to blue screens and buffer overruns. It’s not just developers writing new bugs, or software specifications that require "testing" per se, but grocery stores with mismatched or missing prices, misprinted ferry schedules, voter miscounts, etc. The skill errors are forgivable - when an infielder misplays a ground ball - but the knowledge errors - throwing to the wrong base - continue to amaze. An example of a knowledge error we tend to make in software development is assuming a new user interface is an improvement - often times it is, but each user has a different "experience" and respecting that diversity is something that needs to be honored in the design phase. To non-engineers, software is not intuitive, and can even be intimidating. What seems like a simple or obvious improvement to a proficient user can be sheer terror to a novice.

DDJ: What is the most interesting bug you have seen?
RS: I don’t know if it was the most interesting, but the most memorable one for me was in my first testing role (COM/OLE for the first version of Windows NT). I ran into a case where a mouse move message was not being redirected correctly if it followed a right button click message. It was tremendously obscure and seemed to happen intermittently, but after a lot of time creating convoluted hardware configurations with the mouse plugged, unplugged, and watching the debugger, the logic error was obvious.

Having spent the last couple years on our book The Practical Guide to Defect Prevention I also have a new appreciation for the simple typo and for the minor off-by-one errors that shift the UI by a pixel and stuff like that - they are only noticeable over time, but can turn out to be huge.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RS: It’s imperative to keep the big picture in mind. Many times, the test team can get so focused on perfection that they miss the obvious things that drive down the users' perception of quality. A large scale test effort might help an API work perfectly and handle error conditions gracefully, and performance is rock solid, but there’s a typo in the intro dialog or an edit box is truncated. Keep the user experience front and center. If a bug exists, but the customer doesn’t run into it, is it really a bug? :) Consider how to prevent the bug from appearing in the first place - invest in defect prevention techniques (such as root cause analysis, failure modeling, customer experience, etc.), partner with development, and monitor user feedback.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RS: It sounds a little odd - but I think testers can get in a box and try to be too thorough. The test matrix continues to explode and it is not usually possible to test everything. Again, step back to look at the big picture, and then assess the risk of missing defects in a certain code path. If a particular block of code hasn’t changed, then maybe it’s OK not to test that for a given release. The use of well known risk analysis, probability, and modeling techniques can help guide the test effort and improve effectiveness.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
RS: "Quality at the source" is a well known phrase that should apply in software development. Manufacturing and assembly line work moved many years ago to empower machine operators or assembly line workers with the ability to improve quality. The parallel for software development is improving the quality of the code as its written. Obviously, things like code reviews and inspections, design patterns, unit tests, etc. are a great start, but again, thinking about the big picture, how do we involve users earlier in the design and coding processes to prevent bugs before they are designed or coded into the software?

DDJ: Is there anything else you would like to say?
RS: Please check out The Practical Guide to Defect Prevention and www.defectprevention.org, and definitely send me any comments, feedback, or suggestions.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink