THE BOOK OF TESTING

Thoughts From a Braidy Tester

February 2008

Wayne Miller is a Supporting Architect with the Enterprise Application Architecture Team at ExxonMobil. Which is to say that he spends his time helping people across his company find ways to solve their testing problems. One day he's advising senior architects on long term testing strategy, the next day he's helping frontline testers with their functional testing, and the day after that he's assisting usability and ergonomics groups innovate in their work. Sounds like fun to me!

Here is what Wayne has to say:

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
WCM: Johanna Rothman wrote something that changed the way I think about quality, “Shipping your product is a business decision, not a quality decision.” This is how I’ve achieved what Kent Beck calls “Ease at Work.” I think testers can get very frustrated with work because they are passionate about making the product the best it can be. The irony of software is that it’s never done, and you can chase this goal until you burn out. As Johanna suggests, once you focus your testing energy into providing a forecast of deployment risk to the stakeholders – you can find peace in your work.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
WCM: I am always surprised how consistently test effort is underestimated in this industry. Fred Brooks said 30 years ago, “…few allowed one-half of the project schedule for testing, but that most did indeed spend half of the actual schedule for that purpose.” I think that’s still very common today. I struggle with understanding the forces that perpetuate it. I believe that just about everyone understands the value of testing early. I believe that just about everyone expects troublesome bugs to appear as the product progresses. Across the industry, software project managers and their teams have years of experience. Why do we – the industry - keep repeatedly underestimating testing effort?

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
WCM: It’s hard to remember my first introduction. I often feel like I’m constantly being introduced to testing. It amazes me how many niche testing fields there are in software development – it’s not just enough to have confidence that the application works anymore (maybe it never was). Once you’ve really gotten involved with it full-time, you see how much there is that you have to look at to certify an app – security, load, endurance, usability – it’s a long list. Maybe that’s why we underestimate it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
WCM: Getting developers to be more rigorous about automated unit testing.

I am a big supporter of the concept of Test-Driven Development (TDD). TDD transforms the health of software and opens the gateway to other kinds of tests and QA. I only have anecdotes, but the projects at ExxonMobil that are more disciplined about their unit tests develop faster and with lower bug rates than the projects that do not.

Talking to vendors, I think it is pretty clear that manual testing is still king in the industry, and that has to change. Our systems are becoming more and more interconnected; we have to be able to reliably test these systems as quickly and efficiently as possible.

I think the prize in Service Oriented Architecture demands disciplined test automation to succeed. I don’t believe that manual testing is going to keep pace managing large interconnected systems that are being modified by many people across many locations and groups. It’s not enough that some of our software teams do this well; I think this is a practice that we need to spread consistently throughout our organization.

DDJ: Is there anything else you would like to say?
WCM: One of the new things our testing groups are doing is to start to take a look at the ergonomics of our applications and workflows.

Today, new hires in their early 20s come to work with 15 years or more of wear and tear on their wrists, neck, and back. We want to ensure our employees can maintain the intensity and enjoyment they have for computing when they are 51 or 61 that they had when they were 21.

I’ve appreciated that ExxonMobil takes this commitment seriously and extends beyond just our own internally developed applications. Ergonomics and usability of our applications is a fundamental priority of senior management. We’re working with vendors to develop more usable, ergonomic applications. We continue to extend our network of internal and external ergonomic experts to devise ways to make literally healthier applications and environments for our users.

For us, there is a lot to do and to learn, but developing more ergonomic applications is a win for everyone. Developers take a lot of pride in the effort they put into their applications. They want feedback that gives their applications graceful UIs. Users love transparent applications that allow them to focus on their business problem and forget the tool. Everybody wins from intuitive, integrated applications where no one gets hurt.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Ross Smith has held a variety of jobs in his life: cartooning, guarding jails, managing unions, and that voice on the other end of the line when you call Microsoft Product Support. Then he found testing and hasn't looked back since. As a Tester, Test Lead, Test Manager, Test Architect, and now Director of the Windows Core Security Test Team, he has been involved with Windows and Office for twelve years running. His recently published The Practical Guide to Defect Prevention is one way he aims to share all that experience with the rest of us. (See my review elsewhere on DDJ.)

Here is what Ross has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
RS: It’s surprising to me how often people make mistakes. Human error is a fascinating topic - from nuclear disasters and bridge collapses to blue screens and buffer overruns. It’s not just developers writing new bugs, or software specifications that require "testing" per se, but grocery stores with mismatched or missing prices, misprinted ferry schedules, voter miscounts, etc. The skill errors are forgivable - when an infielder misplays a ground ball - but the knowledge errors - throwing to the wrong base - continue to amaze. An example of a knowledge error we tend to make in software development is assuming a new user interface is an improvement - often times it is, but each user has a different "experience" and respecting that diversity is something that needs to be honored in the design phase. To non-engineers, software is not intuitive, and can even be intimidating. What seems like a simple or obvious improvement to a proficient user can be sheer terror to a novice.

DDJ: What is the most interesting bug you have seen?
RS: I don’t know if it was the most interesting, but the most memorable one for me was in my first testing role (COM/OLE for the first version of Windows NT). I ran into a case where a mouse move message was not being redirected correctly if it followed a right button click message. It was tremendously obscure and seemed to happen intermittently, but after a lot of time creating convoluted hardware configurations with the mouse plugged, unplugged, and watching the debugger, the logic error was obvious.

Having spent the last couple years on our book The Practical Guide to Defect Prevention I also have a new appreciation for the simple typo and for the minor off-by-one errors that shift the UI by a pixel and stuff like that - they are only noticeable over time, but can turn out to be huge.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RS: It’s imperative to keep the big picture in mind. Many times, the test team can get so focused on perfection that they miss the obvious things that drive down the users' perception of quality. A large scale test effort might help an API work perfectly and handle error conditions gracefully, and performance is rock solid, but there’s a typo in the intro dialog or an edit box is truncated. Keep the user experience front and center. If a bug exists, but the customer doesn’t run into it, is it really a bug? :) Consider how to prevent the bug from appearing in the first place - invest in defect prevention techniques (such as root cause analysis, failure modeling, customer experience, etc.), partner with development, and monitor user feedback.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RS: It sounds a little odd - but I think testers can get in a box and try to be too thorough. The test matrix continues to explode and it is not usually possible to test everything. Again, step back to look at the big picture, and then assess the risk of missing defects in a certain code path. If a particular block of code hasn’t changed, then maybe it’s OK not to test that for a given release. The use of well known risk analysis, probability, and modeling techniques can help guide the test effort and improve effectiveness.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
RS: "Quality at the source" is a well known phrase that should apply in software development. Manufacturing and assembly line work moved many years ago to empower machine operators or assembly line workers with the ability to improve quality. The parallel for software development is improving the quality of the code as its written. Obviously, things like code reviews and inspections, design patterns, unit tests, etc. are a great start, but again, thinking about the big picture, how do we involve users earlier in the design and coding processes to prevent bugs before they are designed or coded into the software?

DDJ: Is there anything else you would like to say?
RS: Please check out The Practical Guide to Defect Prevention and www.defectprevention.org, and definitely send me any comments, feedback, or suggestions.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Matthew Heusser develops software, such as a financial application that handles hundred of millions of dollars of fund transfers each year. Matt tests software too, where "testing" might mean writing unit tests, manually poking at an application, or driving the release process for a commercial application which brought in millions of dollars in revenue. And Matt also teaches other people how to develop and test software effectively. You may have read one of his articles here on DDJ Online, or heard him speak at any of a multitude of conferences, or seen him on the SW-IMPROVE discussion list he co-founded.

Matt understands that you may not agree with what he has to say, so he offers only two guarantees when you talk with him: that you'll leave the conversation thinking, and that you won't have been bored in the meantime.

Here is what Matt has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MH: Early in my career, I worked for a company that demoed some software and had an instant sale.

They pushed the demo into production, then brought in the new hire for "maintenance." I was the new hire.

We had no requirements process, no change control process, no version control, no code review, nothing. Oh, and the software didn't work in the first place. All I knew was poke-testing, also known as happy-path testing.

The results, though tragic, were predictable - and painful. I thought to myself "There has got to be a better way" and started learning. Along the way, I found that I actually enjoy software testing; that this task many people think of as dull and boring, when done well, was anything but.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MH: The huge disparity between excellence and mediocrity. A few years ago, Brett Pettichord came up with his concept of "Four Schools of Software Testing", but I believe he missed the biggest one - the oblivious school. So many people in the field today have little knowledge about testing and no interest or incentive to learn. (No, I am not worried about offending them; they don't read blogs.)

I remember when Extreme Programming first became popular, and people like Ron Jeffries and Kent Beck were claiming that testing roles were ineffective and obsolete. I thought to myself "To think that, you must have never met an effective software tester" and that is probably true. They hadn't. The only interaction they had had were with members of the oblivious school.

It amazes me when people talk about test cases run in a day and they count single digits. Of course, that is possible - IF you spend four hours a day in meetings, and IF you create detailed test cases according to the IEEE 829 spec, if you use no automation, and if you use no exploratory or rapid methods.

When I run informal testing challenges at conferences, among my colleagues, if someone can't run ten test cases in three minutes, I'm surprised.

DDJ: What is the most interesting bug you have seen?
MH: The Microsoft Paint Image->Attributes->Height 1, Width -1 bug.

It's interesting because it's an unsigned to signed rollover (didn't those go out with thin ties in the early 1990's?), that it is in a mature product, and that it is probably not worth fixing. It's not worth fixing for a couple of reasons - First, Paint is a giveaway product that demos the features of windows. If you want something solid, get Photoshop or anything else. Second, there is a clear work around; -1 just plain isn't valid; the people that trip it are testers. And third, I hope, it's a great bug for testers when they do demos of exploratory testing.

In that case, I hope Microsoft keeps it around a long time.

DDJ: How would you describe your testing philosophy?
MH: Great Question. My short answer is that when testing is actually running the program under test ("dynamic testing"), you are going through a sort of investigative process of "will this software meet my customer's needs."

Unless the software is extremely simplistic, that turns out to be a rather complicated question to answer. We can never know for sure - there is always one more test to run. So "Are we done answering the question yet?" turns out to be another tough one. Then there's "What do we know with the research we have done so far?"

So I see testing as answering these really tough questions in a creative way. Any tool we can use to accomplish that - scripting languages, driving the browser, FIT, jUnit, Ruby, Windows Explorer, Task Manager, Excel Macros, VBA - any tool we can use - may be helpful.

I don't think you have to be a computer programmer to be an effective tester; I would hold critical thinking as the key skill. I've known many testers who didn't know the testing literature but were wildly effective - in every case, they were extremely strong critical thinkers. Having a CS degree just gives you the ability to do some things *really* *fast* and, in some cases, at all.

I generally split tests into two overlapping groups - developer facing tests and customer facing tests. The Dev tests are TDD and Automated Unit Tests - the Customer-Facing tests are what we now call "acceptance tests", but also "traditional" black box testing. For developer facing tests, I like the idea of a standard regression suite that makes sure we didn't break anything as we change our code. The Pavlovian response to the green bar - "oh, good, I green barred, I can go to sleep now" worries me in some cases however, because in that whole test "cycle", the person didn't do any critical introspective - any critical investigation - all the stuff I defined as testing above.

Also, I don't know if your readers are familiar with the mine field testing analogy, but I'm less and less in favor of running the exact same tests every time. Two common approaches that avoid that are model-based testing and exploratory testing. I kind of run down the middle; I use whatever automation tools that I find valuable in the moment to help me determine the state of the software under test.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MH: I think there are a lot of "red herrrings" out there about software testing. The first is probably the cult of requirements. Look, we've been saying we need better requirements for twenty years; that strikes me as placing testers in the position of the victim. "Get me better requirements" / "I need to be involved up front" / "I need a seat at the table" / "I need management buy in" / whatever. All of those are good things, but essentially, as an industry, we are insisting that other people change, not us. At a recent lunch conversation with Michael Bolton, he said that if he got involved early, that's great, but if not - he still wanted to do the best possible job with the time and resources he was given. I respect that stance.

We need to spend more time talking about how to improve our own skills, and less time asking other people to change to make our jobs easier.

Another one is the cult of documentation, which says that every test case must be documented in excruciating detail, then reviewed. One thing I know: If you spend eight hours a day documentin', one thing you ain't doin' is testing the software.

On the flip side, one thing I think we should *not* ignore, that *is* important, is writing skill (and, of course, testing skill, but I think I already mentioned that). Spending eight hours documenting is bad, but it's worse if you don't have the writing skill to make what you are reading compelling - in that case, nobody's going to read it. Or if you don't have the ability to sift the wheat from the chaff.

It's really easy to document the wrong thing. Then you didn't just waste your time - you've wasted the time of everyone who will ever read your document.

DDJ: Is there anything else you would like to say?
MH: Don't Belong, Never Join, Think For Yourself - Peace. No, wait, that's Victor Stone's Tag Line. Bummer.

I should add, probably to the Philosophy section:

It is so easy to misconstrue and miscommunicate things in the world of software testing. For example, I wrote that we answer questions like "Does the software work?"

Of course, we can't answer a question like that. Instead, on our best days, we can give a rough estimate of the state of the software under test. Likewise, we can't answer "Are we done?" - instead, we can provide information for decision makers, and let them decide whether to take one calculated risk (shipping buggy software) or, possibly, another (missing a market window).

That all depends on our mission - bug hunting vs. assessing quality are two very different missions. However, some missions are better than others. For example, the mission of "verify the software is correct", when the application has anything more than trivial bugs, is a pretty fruitless one. How can you verify a property that is not true?

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |