Five Questions With Elisabeth Hendrickson

I first met Elisabeth Hendrickson several years ago at SDWest, where she gave a workshop on testing. When she asked who in the audience was a tester, I was the only person who raised my hand. Then she asked me what I was doing there - she didn't think I would learn anything, I suppose. Which turned out to be patently not true, as I had expected. I especially like her Nouns And Verbs technique for generating test cases. And her Nightmare Headline Game. And...well, rather than enumerating every last one of Elisabeth's ideas, I'll simply point you to her Ruminations blog so that you can find your own favorites.

If Elisabeth's blog whets your appetite for some more personal assistance with testing, you can catch her at SD and other conferences. Or she would be happy to take your money and work with you in the flesh. (Only please don't introduce her to your workmates as "the expensive consultant from California who used to work for a company whose software sucked", as one manager did!) Here is what Elisabeth has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
ESH: My very first introduction to software testing was at a tiny startup in the 1980s. I was a student at the time, and I’d been hired to hunt bugs. That was the company’s test strategy: hire students and pay them a bug bounty for every bug they found. One problem: I didn't know what I was doing. I didn’t understand the technology or the domain, and I knew nothing about testing. I think I found one bug in the whole time I worked there. I don’t think the company knew what they were doing either (ultimately, like many startups, the company went out of business). So when I cite my testing experience, I don't usually count that particular job.

I did learn one thing on that job that stayed with me. I was trying to do something in the system, and I couldn't. "Oh," I thought. "I must just not have sufficient permissions." Later, I discovered that permissions had nothing to do with it. The problem was with a setting in the operating environment.

That experience taught me to look for multiple possible causes for any given observed behavior. I can't assume that my first guess about a problem is right.

Unfortunately, like many lessons I needed to learn, I had to relearn that lesson many times before it stuck.

Some years after that experience, I was one of the more senior testers in yet another startup. In one particular bug report, I included a guess about the underlying problem in the code. A week later, one of the more junior programmers on the project approached me cautiously. "Elisabeth," he said. "You indicated in this bug report that the problem is in the database connection library. I've been all through that code, and I can't see the problem. Could you show me?" The programmer was quite shy about asking. I think he was afraid to look stupid.

I felt terrible, and I was the one who should have been more concerned about looking stupid. I'd made an assertion about the underlying problem without any real evidence. I guessed. It was totally irresponsible on my part, and I'd wasted days of this guy's time. Once again, I'd fallen into the trap of assuming that my first impression about an underlying cause was accurate. The problem turned out to be somewhere completely different.

I hope the lesson has finally stuck, but even now I have to be vigilant to make sure I don't repeat the error.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ESH: The most important thing for any tester to do is connect with their stakeholders: the people who will use the information that testing reveals to make decisions about the project and the software.

I once met a group of testers who had no idea why they tested. They just did. They'd been hired to test, so they tested. They wrote test reports no one ever read and filed bug reports about defects no one ever fixed. They were a very frustrated group of people. They didn't know why no one seemed to be listening to them. When I suggested they talk to their stakeholders, they realized that they didn't even know who in the organization should care about their work.

We talk a lot about requirements for software. Few testers think through the requirements for their testing. And yet there are requirements for testing. There are pieces of information the stakeholders need to have and that testing can provide.

Consider the VP in charge of deciding whether or not to ship the software. She needs to know if the software does what she promised the customer it would do. She needs to know that it doesn't do anything that would harm the customer. And she needs to know that the tests were sufficiently comprehensive that they would have revealed any problems if they existed.

A test group that focuses on reporting bugs won't be able to tell this VP everything she really needs to know. Bug reports tell us what isn't working, but don't tell us anything about what is working.

A test group that focuses on test status reports with pass/fail information won't be able to give this VP the information she needs either. Test pass/fail statistics are usually completely meaningless by themselves. What does it mean to say "We ran 543 tests on the Foo feature with a 87% pass rate?"

To serve this VP well, the test group has to work with the VP to understand what information she'll value, how often she wants to receive it, and what format she'd like to see it in.

And the test group also has to realize that this VP is only one of their stakeholders. The developers also need information from them. Technical support may need information as well. And the technical writers. We testers are in the information business, we have numerous "customers" for the information we provide, and we should do our best to delight our customers with the quality, quantity, frequency, and overall usefulness of the information we provide them.

Knowing about system analysis, test design, test automation, or about the particular domain are all fine and well. But testers who don't know what information their stakeholders need will find that all those other skills don't add up to a whole lot.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
ESH: I think that "repeatability" is overrated. I started hearing the "Testing should be repeatable" mantra in the late 1980s, along with "Testing should be independent" and "If it isn't documented it didn't happen," and other great sayings that work better on a bumper sticker than in reality.

The "repeatability" requirement for testing particularly bothers me because I can think of so few situations in which repeatable is the most important attribute for a given set of tests. I'd rather tests provide useful information about the capabilities and limitations of the software than that they be repeatable.

I think repeatability became so important because it increases manageability. If you execute the same 5246 test cases on every build, it's easier to predict how long testing will take, it's easier to compare the results of two test runs, and it's easier to track progress against a plan. The problem is that none of those things make software better. They may make the process more predictable and measurable, but they don't make the software better. More information can make software better. So I think it's more important that tests be informative than repeatable.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ESH: Learning how to integrate their efforts with the rest of the team. It's not as easy as it sounds.

Traditional testing practices involve designing tests in parallel with the software development, not in collaboration. Testers are accustomed to working on test design, documentation, and execution in isolation. Some have become comfortable with that isolation. They like being able to work independently. They prefer not to have to explain in great detail how they designed a given set of tests.

The problem with this approach is that too often testers end up re-inventing the wheel...or, in our field, re-inventing the structured programming language. We waste time reverse engineering the software to figure out how to test it. We struggle to build test automation exclusively through external interfaces that weren't designed with test automation in mind. We spend time testing conditions the team does not care about and miss cases that the team does care about.

I'm a member of the Agile community and a huge fan of Extreme Programming and Scrum. Agile teams work as a whole, integrating their efforts, and collaborating on the end result. On Agile teams, QA/Test works with the team to move the project forward rather than gate its passage. That's a huge mindshift. Testers on Agile teams have to learn to provide feedback instead of criticism and to apply their testing skills much earlier in the process.

Some have expressed their concern to me that the "integrated team" approach means that testers have to learn to code. I don't think that's true. I think that testers have to bring something to the table that the team values. The developers already know how to code, so someone who specializes in testing probably isn't going to add much in the way of development expertise. Instead, the testers should bring domain expertise and test expertise to the party.

On integrated teams, testers help the other team members learn more about the domain and about testing. They do more than test: they help the team define acceptance criteria, collaborate with developers and business stakeholders to define tests, collaborate with developers to automate the tests, and support the team as a whole by helping the team figure out what "done" looks like and how they'll know when they've achieved it.

Not all testers are comfortable taking such a proactive role, and that's why being a full fledged member of an integrated team is a big challenge.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
ESH: You should ask me "Why do so many people consider testing tedious, and why do you think it's fun?"

People think testing is tedious when they see it as a monotonous process of repeating the same actions over and over. Click this button, enter that value, check for this error. It's repetitive. Boring. Mind numbing.

I have a different perspective. I think that if your tests are that repetitive, that boring, that predictable, you should have automated those tests some time ago. Let computers do what computers are good at: repeating sequences over and over. And let humans do what humans are good at: challenging, analytical, creative work.

As a tester, I see the big picture, and I also have to be able to drill into the details. I examine the system from the outside, and it also helps if I understand the system from the inside. Testing requires me to think about how the software will be used, what kind of vulnerabilities it might have, and what might be interesting things to vary. And that means testing gives me the opportunity to use a wide variety of skills: technical skills, analysis skills, communication skills, and organizational skills. As a tester, software feels like a maze to be explored, a puzzle to be solved. I love the intellectual challenge.

Posted by The Braidy Tester at 07:30 AM  Permalink