Five Questions With David Treadwell

David Treadwell joined Microsoft two months after I graduated from high school. I don't know whether that makes him really old or me really young. <g/> Back in the day he wrote file server and TCP/IP code for Windows NT. Perhaps his most visible contribution to date, however, has been running the .Net Developer Platform team, otherwise known as the people who invented and developed Microsoft's .Net Framework. While there he shepherded .Net 1.0's creation and release and helped .Net 1.1 get out the door. About a year ago he moved over to the Windows Live Core team, where he is working on all sorts of nifty secret things which we will hopefully learn about before too long. Here is what David has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DT: I’m going to take the liberty to give a two-part answer.

My first formal introduction to testing was a course I took in college. I was an electrical engineering major, and one of the senior-year courses was about silicon chip testing. Much of the course was about things like error-correcting codes and the like, which are not especially relevant to software testing. However, one principle from the course really stood out to me: the value of designing for testability. The course strongly made the point that one must consider testing early in the design phase of an engineering project because the cost of testing can be dramatically reduced if the design is built for testability.

The other major early exposure to test for me was in 1992 when I was developing the NT Winsock implementation. I had actually been at Microsoft for several years working on the SMB server, and while I had a fair amount of interaction with test, the relationship was somewhat distant. What changed when I was working on Winsock is that I was pretty much the sole developer responsible for the project, and the company had assigned exactly one tester to work on the project with me: Jim Gilroy.

At first, the way things worked is that Jim would write some code to invoke the Winsock APIs that I had written, then he would file bugs if he thought he found problems. I would typically find these bug reports annoying for a variety of reasons: sometimes they were outright bogus, sometimes they were of questionable legitimacy, and sometimes they were real and pointed out my own failings. Whatever it was, I usually felt like addressing these bugs was a distraction from the “real” work of writing new code.

Before too long, however, my code started getting out into the wild via early developer beta releases of NT. Real external people were using my code to write real apps. Predictably, these apps sometimes ran into real problems in my code. These problems were always VERY expensive to address, since I often did not have the app code or easy reproducibility of the issue. It was also very annoying to the people trying to write apps on top of this very new infrastructure.

Eventually, something that now seems so obvious just clicked for me: Jim was not an adversary. He was a key partner to me. If Jim’s test harness found problems in my code it was VASTLY easier for me to address the issue than if a customer ran into the problem. I came to appreciate just how selfishly valuable this resource was to me, and it changed my whole perspective on the test relationship. I listened much more to the suggestions that Jim made. I proactively gave him suggestions on things he should include in his tests. I spent a lot more energy looking into things before assuming it was a test bug. Through it, Jim was patient with me and the process, always focused on making the product better.

Jim’s tests eventually became the de facto standard for the manner in which a Winsock implementation should behave, and this was a small but very important part of the growth of the Internet. Without Winsock as a common platform on which to write applications like Mozilla, it would have taken much longer for the Internet to change society in the way it has.

My lesson from all of this: test is a key ally in building a great product. Treat test as a partner, not an adversary.

DDJ: What is the most interesting bug you have seen?
DT: Back in 1991, when working on the NT SMB (file) server, we were starting to do “dogfooding” of our system. We stored the NT source code on systems running super early versions of NT—essentially daily builds. These would crash on occasion, which was annoying to folks who wanted to synch or check in code. However, what was really bad was when there was corruption of the source files. When someone broke the build, there was appropriately a lot of negative feedback given to the person. However, when the person had checked in good code, but the server had “munged” it, people got outright pissed.

We had one data corruption bug in these dogfood source servers that was rare but persistent. Eventually, my boss asked me to drop everything until I got this bug fixed. Because it was uncertain whether the bug was in the server code or somewhere else in the system, management also assigned another developer from the core OS team, Mike Glass, to work on the issue with me.

Mike and I spent several weeks trying to instrument the system to track down the problem. Because the problem rarely reproduced, and because we only learned of corruption well after the bug occurred, it was a total nightmare to track it down. It was bad enough that I had dreams about memory patterns and assembly traces. Eventually, Mike tracked down the bug to a problem in the SCSI hardware that we had been using on those systems. Pascal Zachary mentions this story in his book Showstopper.

Lessons: do as much instrumentation as possible, tools matter, and be open-minded and persistent about where the problem may lie.

DDJ: How would you describe your testing philosophy?
DT: I largely described it in the story above about Jim Gilroy: test is an essential part of the software engineering process. Good, efficient test goes a very long way toward making a product great.

A corollary to this philosophy is that automated testing is much more powerful than manual testing. When things are automated, it is straightforward to put new code or changes “through the wringer” to ensure that it still works—or to find issues in it.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DT: For the tester: be patient with your developers who might not yet see your value. If they are any good, they will come around eventually. Also, be tenacious about being the quality gate for the product. Do not expect perfection in the product, but assertively demand excellence. Think hard about the real scenarios that users will put the product through, and gauge the importance of your bugs based on that. Not everything is a P1, but when you find something that will truly be a real negative impact on users of the product, hold the team’s feet to the fire on getting it addressed.

For the developer: those testers may seem annoying, but understand that, in the long run, they are some of the best partners you have. You should MUCH prefer that they find the bugs instead of your customers.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DT: The growing importance in the industry of services will bring some interesting challenges and opportunities to the test discipline. The challenges come from the fact that services have very different processes and constraints than typical packaged software: essentially continuous release cycles, 24x7x365 operation of the service, massive scale data centers. The opportunities come from the fact that services have some nice agility advantages over packaged software, and these advantages should allow more efficient testing in many dimensions.

Posted by The Braidy Tester at 07:30 AM  Permalink