Five Questions With Eric Sink

Eric Sink has been on my Must Read list for years now. His series on the business of software for MSDN - now collected into a most excellent book titled, appropriately enough, Eric Sink on the Business of Software - is the place to start; if you like what you read, head on over to his Eric.Weblog() for much, much more. Or visit Joel On Software's Business Of Software forum, which Eric moderates.

Back in the day Eric founded the AbiWord project and was a project lead at Spyglass when they were designing the original versions of the Browser-Currently-Known-As-Internet-Explorer. Today he is a Software Craftsman at SourceGear. Here is what Eric has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
ES: I think I am continually surprised how hard testing is.

One goal of testing is to find (and fix) bugs before our customers do. Our products go out to tens of thousands of users. That's considered a very, very small customer base when you remember that Microsoft's products go out to hundreds of millions of users. Still, even at our scale, those users represent an astonishing diversity of configurations and usage styles.

Just to use round numbers, think of the problem like this: You have a team of ten people and their computers. They are trying to find all the bugs that would be found by a group of ten thousand people and their computers. The only advantage is that the former group is specifically looking for the bugs while the latter group is specifically hoping not to encounter them.

It's an amazingly hard problem.

DDJ: What is the most interesting bug you have seen?
ES: My favorite bug happened early in my career while working on scientific visualization tools at Spyglass.

It all started when a user called our tech support team and said, "I was using your product and MPW crashed." (MPW stands for Macintosh Programmers Workshop, Apple's development IDE back in that day.)

Our first reaction was to assume that MPW's crash had nothing to do with us. I mean really. Why blame us because MPW crashed? He was using lots of other things at the same time. Did he call his electric company and tell them he was using the power grid when MPW crashed?

So we asked all the usual questions. Does MPW always crash when using our product? No. How often does it happen? I don't know.

We concluded that there was no connection. Our customers were mostly scientists. We were accustomed to getting some very strange phone calls.

But this particular situation happened again with somebody else. And then somebody else. And it kept happening. Eventually we had to conclude that the problem was real. But since it was completely non-reproducible, we had no idea how to fix it.

I still remember my manager boasting that if anyone could ever make the bug reproducible, he would fix it that day. And one day, that's exactly what happened. A user called in with a recipe to reproduce the bug. It had to be followed exactly, and it had to be done on a certain kind of Mac. At the end of the sequence of steps, boom, MPW crashed.

And my manager found and fixed the bug within a couple hours. We were forgetting to free a palette handle.

DDJ: How would you describe your testing philosophy?
ES: My philosophy about testing is all about balance. Some kinds of testing seem to produce more bang for the buck than others, but the best results happen when multiple kinds of testing are used. Testing is not an area where it pays to put all your eggs in one basket.

My own particular interest is in automated testing, especially when used with code coverage. My pet peeve is people who argue against automated testing by saying that it's not enough. For some reason, people get themselves in the mindset that automated testing is only worth the trouble if it becomes sufficient for all their testing needs. Well of course automated testing is not enough! Nobody said it was or should be, but there are darn good reasons for doing it anyway. It's all about balance.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ES: I personally think everything works better when we all agree that the job of testing is not to ensure or verify the quality of the software, but rather, to measure the quality of the software. This is a subtle distinction, but an important one.

In the course of shipping a product, good testing will find lots of "bugs" that will not be fixed. I argued this view in an article on my blog entitled "My Life as a Code Economist". For anyone with real experience shipping a product, this is obvious. For anyone else, this is heretical.

We don't want testers to make sure there are no bugs. We want testers to tell us about the bugs that we've got so we can make an informed and wise decision about what to do with them.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ES: Bug tracking tools are growing up, and fast. We have traditionally thought of bug-tracking as a very simple tool. However, bug-tracking tools are getting more and more integrated with other team development tools like version control and builds. The rate of change in the toolset promises to keep our lives interesting for some time to come.

Posted by The Braidy Tester at 07:30 AM  Permalink