Five Questions With Matthew Heusser

Matthew Heusser develops software, such as a financial application that handles hundred of millions of dollars of fund transfers each year. Matt tests software too, where "testing" might mean writing unit tests, manually poking at an application, or driving the release process for a commercial application which brought in millions of dollars in revenue. And Matt also teaches other people how to develop and test software effectively. You may have read one of his articles here on DDJ Online, or heard him speak at any of a multitude of conferences, or seen him on the SW-IMPROVE discussion list he co-founded.

Matt understands that you may not agree with what he has to say, so he offers only two guarantees when you talk with him: that you'll leave the conversation thinking, and that you won't have been bored in the meantime.

Here is what Matt has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MH: Early in my career, I worked for a company that demoed some software and had an instant sale.

They pushed the demo into production, then brought in the new hire for "maintenance." I was the new hire.

We had no requirements process, no change control process, no version control, no code review, nothing. Oh, and the software didn't work in the first place. All I knew was poke-testing, also known as happy-path testing.

The results, though tragic, were predictable - and painful. I thought to myself "There has got to be a better way" and started learning. Along the way, I found that I actually enjoy software testing; that this task many people think of as dull and boring, when done well, was anything but.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MH: The huge disparity between excellence and mediocrity. A few years ago, Brett Pettichord came up with his concept of "Four Schools of Software Testing", but I believe he missed the biggest one - the oblivious school. So many people in the field today have little knowledge about testing and no interest or incentive to learn. (No, I am not worried about offending them; they don't read blogs.)

I remember when Extreme Programming first became popular, and people like Ron Jeffries and Kent Beck were claiming that testing roles were ineffective and obsolete. I thought to myself "To think that, you must have never met an effective software tester" and that is probably true. They hadn't. The only interaction they had had were with members of the oblivious school.

It amazes me when people talk about test cases run in a day and they count single digits. Of course, that is possible - IF you spend four hours a day in meetings, and IF you create detailed test cases according to the IEEE 829 spec, if you use no automation, and if you use no exploratory or rapid methods.

When I run informal testing challenges at conferences, among my colleagues, if someone can't run ten test cases in three minutes, I'm surprised.

DDJ: What is the most interesting bug you have seen?
MH: The Microsoft Paint Image->Attributes->Height 1, Width -1 bug.

It's interesting because it's an unsigned to signed rollover (didn't those go out with thin ties in the early 1990's?), that it is in a mature product, and that it is probably not worth fixing. It's not worth fixing for a couple of reasons - First, Paint is a giveaway product that demos the features of windows. If you want something solid, get Photoshop or anything else. Second, there is a clear work around; -1 just plain isn't valid; the people that trip it are testers. And third, I hope, it's a great bug for testers when they do demos of exploratory testing.

In that case, I hope Microsoft keeps it around a long time.

DDJ: How would you describe your testing philosophy?
MH: Great Question. My short answer is that when testing is actually running the program under test ("dynamic testing"), you are going through a sort of investigative process of "will this software meet my customer's needs."

Unless the software is extremely simplistic, that turns out to be a rather complicated question to answer. We can never know for sure - there is always one more test to run. So "Are we done answering the question yet?" turns out to be another tough one. Then there's "What do we know with the research we have done so far?"

So I see testing as answering these really tough questions in a creative way. Any tool we can use to accomplish that - scripting languages, driving the browser, FIT, jUnit, Ruby, Windows Explorer, Task Manager, Excel Macros, VBA - any tool we can use - may be helpful.

I don't think you have to be a computer programmer to be an effective tester; I would hold critical thinking as the key skill. I've known many testers who didn't know the testing literature but were wildly effective - in every case, they were extremely strong critical thinkers. Having a CS degree just gives you the ability to do some things *really* *fast* and, in some cases, at all.

I generally split tests into two overlapping groups - developer facing tests and customer facing tests. The Dev tests are TDD and Automated Unit Tests - the Customer-Facing tests are what we now call "acceptance tests", but also "traditional" black box testing. For developer facing tests, I like the idea of a standard regression suite that makes sure we didn't break anything as we change our code. The Pavlovian response to the green bar - "oh, good, I green barred, I can go to sleep now" worries me in some cases however, because in that whole test "cycle", the person didn't do any critical introspective - any critical investigation - all the stuff I defined as testing above.

Also, I don't know if your readers are familiar with the mine field testing analogy, but I'm less and less in favor of running the exact same tests every time. Two common approaches that avoid that are model-based testing and exploratory testing. I kind of run down the middle; I use whatever automation tools that I find valuable in the moment to help me determine the state of the software under test.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MH: I think there are a lot of "red herrrings" out there about software testing. The first is probably the cult of requirements. Look, we've been saying we need better requirements for twenty years; that strikes me as placing testers in the position of the victim. "Get me better requirements" / "I need to be involved up front" / "I need a seat at the table" / "I need management buy in" / whatever. All of those are good things, but essentially, as an industry, we are insisting that other people change, not us. At a recent lunch conversation with Michael Bolton, he said that if he got involved early, that's great, but if not - he still wanted to do the best possible job with the time and resources he was given. I respect that stance.

We need to spend more time talking about how to improve our own skills, and less time asking other people to change to make our jobs easier.

Another one is the cult of documentation, which says that every test case must be documented in excruciating detail, then reviewed. One thing I know: If you spend eight hours a day documentin', one thing you ain't doin' is testing the software.

On the flip side, one thing I think we should *not* ignore, that *is* important, is writing skill (and, of course, testing skill, but I think I already mentioned that). Spending eight hours documenting is bad, but it's worse if you don't have the writing skill to make what you are reading compelling - in that case, nobody's going to read it. Or if you don't have the ability to sift the wheat from the chaff.

It's really easy to document the wrong thing. Then you didn't just waste your time - you've wasted the time of everyone who will ever read your document.

DDJ: Is there anything else you would like to say?
MH: Don't Belong, Never Join, Think For Yourself - Peace. No, wait, that's Victor Stone's Tag Line. Bummer.

I should add, probably to the Philosophy section:

It is so easy to misconstrue and miscommunicate things in the world of software testing. For example, I wrote that we answer questions like "Does the software work?"

Of course, we can't answer a question like that. Instead, on our best days, we can give a rough estimate of the state of the software under test. Likewise, we can't answer "Are we done?" - instead, we can provide information for decision makers, and let them decide whether to take one calculated risk (shipping buggy software) or, possibly, another (missing a market window).

That all depends on our mission - bug hunting vs. assessing quality are two very different missions. However, some missions are better than others. For example, the mission of "verify the software is correct", when the application has anything more than trivial bugs, is a pretty fruitless one. How can you verify a property that is not true?

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink