THE BOOK OF TESTING

Thoughts From a Braidy Tester

June 2006

"I think I'm out of my depth here. If you're willing to help me through this, I promise to pester you to pieces with lots of questions and come up to speed just as fast as I can."

When they heard this the team broke out in great big smiles. While it would have been nice to have someone who already understood how things worked, Hazim being willing to learn was the next best thing.

Bianca started things off. "The most important thing, and what we always start with, is understanding what exactly it is our customer wants to do, how they will want to use our application."

"Isn't that what I said? Hazim asked.

"Well, yes, kind of," Jason replied, "but the way we approach it is very different from what you are used to. We used to work that way - build a spec that went into great detail about the UI, and the specifics of how everything worked, and all that. But we found we often missed some core scenarios."

"And we spent way too much time drafting detailed test cases that were obsolete before we finished because our understanding of the application was evolving so rapidly" Daphne added. "Or if we waited to plan test cases until the spec was finalized, Lucas and Bianca were already starting to code and we were behind the game."

"So what we do now", Oliver took over, "is focus on our customer's goals, on the actions they will be executing. These actions don't have anything to do with the user interface, or the programming model, or the multiple ways they can do something. We focus on the core goals our customer is trying to achieve."

"Let's see if I understand" said Hazim. He connected his tablet to the team's giant wall display. "Here's the information marketing sent me regarding what this Rug Rat Rubbings application needs to do."

• Select a page from a coloring book and color it in.


• Select a blank page and create a drawing from scratch.


• Save the current drawing to the child's picture gallery.


• Open a previously saved drawing from the child's picture gallery.


• Print the current drawing.

"So our user actions would be something like this?" Hazim inked the following below the info from marketing:

• Open a coloring book


• Browse the current coloring book


• Select a page from the current coloring book


• Color the current page


• Start a blank page


• Save the current drawing


• Open an existing drawing


• Print the current drawing

"That's a good start!" Bianca affirmed. "I see some additional actions though. I bet we'll have more than just one coloring book, so our customers will need to browse through them all." She added "Browse the library of coloring books" to the end of Hazim's list.

"I think 'Color the current page' is too coarse of an action" Lucas said. "Coloring a page usually involves filling in discrete areas with a specific color, but when a kid starts with a blank page they will draw their own lines. So we need at least 'Fill an area' and 'Draw line' actions."

"Oh!" Daphne exclaimed. "We'd better have some way to select different colors! I don't see children being happy with just one color for very long!" She added a "Pick the color to draw with" action.

Just then Simon, MMS's marketing guy, walked past. "Hey Simon!" Bianca yelled. "About this Rug Rats app. Whaddayathink about providing a set of shapes the rug rats can 'stamp' onto their drawing?"

"Oooh, that's a neat-o groovy idea!" Simon thinks the Sixties are cool again, and the entire team agrees that he actually makes bell bottoms look cool.

"Great, thanks, your grooviness!" Bianca responded. She added "Stamp a shape onto the current drawing" to the list.

Hazim was staring off into space. Jason snapped his fingers in Hazim's face. "Yo, Earth to Hazim, Earth to Hazim. Anyone there?"

Hazim blinked back into this reality. "Sorry, I was just thinking about how to start a blank page. Having a button just for that seems like overkill. What if we had a special "Blank Page" page in every coloring book? Or maybe a special coloring book filled with pages with different background patterns? Oh, but I guess that's several levels of detail lower than we're talking about right now..." he trailed off.

Lucas jumped in. "That's a great idea, Hazim! You're right that it's at a lower level of detail than our user actions generally are, but that's OK. We certainly don't want to lose it!" Lucas started a new page in the digital notebook, labeled it "Implementation Ideas", then scrawled in Hazim's idea.

Daphne and Jason had been whispering and writing off to the side while Hazim and Lucas were talking. "Here's our test API for R3" Jason said. Daphne activated the new page she and Jason had added on her tablet.

Posted by The Braidy Tester at 07:30 AM  Permalink |

"I know that you all need a detailed specification before Bianca and Lucas can start designing their architecture, and that Daphne and Jason need the spec in order to start planning test cases, and that Oliver needs to know what it is he's designing. I'll be working on that today and tomorrow."

Well, that is what Hazim intended to say, but he didn't get any further than "I know that you all need a detailed specification" before the team chorused "No we don't!"

Hazim blinked. Twice. "You don't?"

Oliver tends to be the least snarky of the bunch, so he jumped in before someone else could make a wisecrack. "Nope! We do things differently around here than you are probably used to. You're used to interviewing customers to learn what they want, recording all that information in specifications, having customers sign off on the specs, passing the specs off to developers who turn them into code, developers passing the code off to testers, testers throwing the code back to developers because it's full of bugs plus it doesn't work the way they interpret the specs as saying it should work, developers and testers batting the code and specs back and forth until upper management decides it's time to ship the thing, at which point it's passed off to the customers who say that it's not what they wanted at all. Am I correct?"

"Well...." That wasn't the way things were supposed to work, but Hazim had to admit that that's more-or-less what usually ended up happening. "Yes. Do you do something different?"

"We sure do!" Oliver replied. "I'll borrow you some books that will explain in detail, but the short version is that we do just-in-time everything: feature design, architecture, coding, testing - everything. We have found that by working closely together and being in continuous communication we don't need nearly as much documentation as you're used to working with."

"And - especially at this point - we don't really care how exactly the application will work" Jason added. "We start by determining what our customer is trying to do. Everything else comes from that."

"Oh. Um. Hrm." Needless to say Hazim was a bit nonplussed right now. He knew that MMS was going to be a big change, but he didn't realize it was going to be this big of a change. Luckily for all concerned, he not only realizes that he's out of his depth but he is willing to admit the fact. (Possibly he has read Jerry Weinberg's Becoming a Technical Leader, Secrets of Consulting, and More Secrets of Consulting.)

"I think I'm out of my depth here. If you're willing to help me through this, I promise to pester you to pieces with lots of questions and come up to speed just as fast as I can."

Posted by The Braidy Tester at 07:30 AM  Permalink |

Now that you have an overview of how thinking in terms of user actions can aid and abet your entire product development process, let's dive in and see how it works in detail.

We'll do this by following a product team as it attempts to build and ship an application. Ultimately I intend to post actual working code for interesting points in this process, but given that Seattle is currently in its too-short-for-my-tastes lots-of-sun season, most of my free time will be spent Trikkeing and so copious amounts of code most likely will not be forthcoming.

Let me introduce our cast, who work play work at Mayhem Maven Studios. MMS creates software for children, chil'uns, Offspring Who Cannot Be Tamed. Their latest program is a drawing program named Rug Rat Rubbings. The team consists of

• Hazim, the program manager


• Lucas, a developer


• Bianca, a developer


• Daphne, a tester


• Jason, a tester


• Oliver, the designer

We join them in their initial meeting to discuss this new product.

Oh, but first: there is user participation in this story! Given the little bit you know about Rug Rat Rubbings so far, what do you think our team's customers will want to be doing? What do you think the user actions are? Post your answers as comments, or email them to me.

"You may be wondering why I called you all together today." Hazim likes to think he is quite droll, but in reality he has only a few jokes that he repeats over and over and over. The team has heard all of them. Multiple times. Too many times. Even though he's only been on the team for a few days.

"Nope!" they reply in unison.

"Hey Hazim, I heard about this 'Humor for People Who Have None' class over at Sopwith Community College; I think they would take you pro bono!" That's Bianca, the snarky one.

Pennies descend on Hazim from the general direction of Lucas and Jason. "Hey look! Now you can afford a new writer!" Daphne interjects. Actually, they're *all* pretty snarky.

"Well. Um. So you may have heard we're launching a new product: a drawing program for kids. The marketing whizzes have named it Rug Rat Rubbings." Except for Hazim - he's just recently joined the company, and his previous job was doing line of business applications for a major corporation. He vaguely remembers having a sense of humor back in his college days, but his time in Corporate America buried it under piles of rules and regulations and grim-faced meetings where people had absolutely no fun. Moving to Mayhem Maven Studios has been rather a large change for him, and he's still in a bit of a daze.

"I know that you all need a detailed specification before Bianca and Lucas can start designing their architecture, and that Daphne and Jason need the spec in order to start planning test cases, and that Oliver needs to know what it is he's designing. I'll be working on that today and tomorrow."

Posted by The Braidy Tester at 07:30 AM  Permalink |

I find the Rapid Testing movement interesting because it is completely opposite how I learned to test yet matches quite well with what I actually do. Testing was taught to me as a process where you:

1. Inspect the specification for your feature.


2. Draw up an exhaustive list of test cases for that feature.


3. Prioritize the test cases.


4. Execute each and every test case in priority order.

Usually my process looks something more like this:

1. Inspect the specification for my feature. Where sometimes the "spec" is actual running code.


2. Play around with the feature for awhile - in my head when I have a spec, with the actual feature when I am looking at existing functionality.


3. Get a sense for what parts of the feature my customers are most likely to use, and for which pieces are most likely to work incorrectly.


4. Start banging the first test area, usually with a general plan of attack and sometimes even with an explicit set of tests.


5. Find bugs or things that make me go "Huh?" or interesting bits of functionality I hadn't noticed before.


6. Test around and dig into those bugs and double-takers and interesting bits to learn more.


7. Look over at my clock because my calendar program went "ding" and realize that it's time to go home.

Sometimes I worry that I'm approaching things all wrong. After all, when I make a decision, aren't I supposed to gather loads of information about it, carefully distribute and balance that information, and then the right answer will just be blindingly obvious? I'm so not doing that! I'm making spur of the moment decisions and running with them. So am I being a bad tester?

Thanks to Sources of Power, by Gary Klein, I can now say that I am making Recognition-Primed decisions. Gary and his research team have interviewed whole bunches of fireground commanders, military commanders, nurses, and the like. You know, people who are constantly making important decisions, like ones that affect whether someone lives or dies. What he has found is that while the formal decision processes can be useful for newbies, these types of situations are far too chaotic and fast-changing for formal processes to be effective. What happens instead is that an expert's experience in a domain enables them to see a situation through prototype-colored glasses - they can see through the specifics of a particular situation into the prototypical situation behind it and thus quickly determine the most typical course of action. Rather than comparing multiple options they pick that first one, run a mental simulation to determine whether it will work in the current situation, and then either select that option if it works or move on to a different option if it doesn't. Thus these experts can quickly find a workable solution rather than be stuck in analysis paralysis. They don't try to find the best option, simply one that will work. Nothing guarantees that their simulation will work, and sometimes they will be wrong. When that happens, or the situation changes, they iterate and move on.

Sounds a lot like testing, no? If this sounds anything like the way you work, I think you will find Sources of Power fascinating. Or if this sounds anything like the way your testers work, this book will help you understand them a little better! <g/>

Posted by The Braidy Tester at 07:30 AM  Permalink |

Recently I was reviewing (i.e., testing) a manuscript. At one point I saw this heading:

(c) Blah Blah Title

"Where's A and B?" I asked in my comments.

Later I saw it again:

(c) Some Section Title

"Huh? What, this section is copyrighted?" I wondered.

And then again:

(c) Titleist Titly Title

Now I was pretty sure there was something going on. "Oh - that heading is three levels in. Maybe the underlying markup is leaking through?"

Finally I saw it again:

(c) Some Title

But this time it was immediately followed by another heading:

(d) Some Other Title

Aha! "Yup, almost certainly markup is leaking through!"

As I continued through the manuscript I came across many more cases of this "bug", each of which further bolstered my theory that markup was leaking through.

It struck me later that this was a good example of the process I go through when I find a bug. First I notice something odd. Then I try to make the bug happen again (in this case, find it recurring in a different location). I continue testing around the bug, eventually uncovering enough information that I can develop a hypothesis regarding what might be going on. And then I do yet more testing just to make sure I really do understand what's going on. And then I report it and ruin a developer's day. <g/>

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless…
You have at least considered testing on the following hardware configurations:

• Low end desktop
• High end desktop
• Low end laptop
• High end laptop
• Minimum expected screen resolution (this may be 640x480 in some cases, but depending on your customers you may be able to expect a higher resolution)
• Super-high screen resolution (yes this is an excuse to get one of those thirty-inch flat panels! <g/>)
• Other relevant form factors (e.g., for Microsoft Windows: convertible Tablet PC, slate Tablet PC, UMPC/Origami, Pocket PC, Smartphone)
• Maxed-out super-high-end-everything machine
• Minimum configuration machine
• Laptop with power management settings disabled
• Laptop with power management settings set to maximize power
• Laptop with power management settings set to maximize battery life
• CPU configurations

Posted by The Braidy Tester at 07:30 AM  Permalink |

I mentioned in the introduction that thinking in terms of user actions can be just as useful for manual testing as it is for automated testing. You might think that a tester exploring an application would naturally do so in terms of user actions, since they are effectively acting like a user. This is not necessarily the case, however: I find it just as easy to get caught up in the mechanics of using an application (that is, to get caught up in testing a particular dialog box, for example) when I am physically poking at it with a mouse and keyboard as when I do so through automated tests.

Certainly all those details need to be covered at some point, but keep in mind what it is your customers are trying to do. Almost certainly people aren't purchasing your application to experience the joy of interacting with a particular palette or wizard, but rather to design a user interface or draw a picture or develop their own application or draft an essay. Focus on what your user wants to do and why they want to do it and you will hit the important *how*s automatically.

Posted by The Braidy Tester at 07:30 AM  Permalink |

As you continue testing the now theoretically-complete feature, track the characteristics of the bugs you find. You'll also start getting bugs from users; track these as well. Examine each of these bugs in terms of user actions: many of the bugs will fit nicely into the user actions you've defined, but other bugs will suggest new actions. Remember that Test code is just like any other code: you write it based on your initial understanding of the problem; as the arrival of new information changes your understanding of the problem it is tremendously helpful to morph that code to be in sync with your new understanding.

Also useful at this point - throughout the process, actually, but especially here - is to perform a mini-retrospective of the process. What went so well you want to be sure to continue doing it? What went so poorly you want to be sure to never do it again? What changes would you like to try? Look at the changes your user actions went through. If those changes were a reaction to changes in application functionality, how could you have foreseen the changes? Was the specification unclear? Or did the feature team have an insufficient understanding of what your customer needed? If you simply missed an entire user action, why did you miss it and how can you be more likely to identify it next time? If you look at what went wrong, identify changes that seem likely to ameliorate those problems, implement the changes, and then repeat the cycle, you can't help but get better.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Now it's milestone time. Regardless of whether your milestones last for two weeks, two months, or two years, (someone on) your team somehow decides what all will fit into the milestone. Once the milestone's features are decided each feature must be described in a specification. The exact form these specs take again depends on your environment, but you always have a spec. User actions come into their own here as you convert each spec into a set of user actions. Can't decide what the user actions for a feature are? Is the team unable to agree on what the user actions are? Clearly the specification needs further clarification.

As (portions of) the feature definition stabilizes your developers will start implementing it. At the very same time you can start writing your test libraries and automated test cases - the user actions you helped your team define become your test APIs. You probably don't know how you will implement them yet, but simply stubbing them out is sufficient for you to start writing test cases against them. Since the user actions don't have anything to say about UI, it doesn't matter if you don't know what the user interface will look like or how it will work. You don't have to know how you will verify the actions either. Your test cases won't run, but just thinking them through at this high level will help you find areas where you don't understand things as well as you thought you did.

Dev implements, implements, implements. They can use your stubbed-out test cases and the user actions on which they are built as a remembery for how the feature is expected to work. As the user interface becomes defined you can implement your test libraries in terms of an object model around that UI - so again you don't need to know exactly where the various bits of UI will be, or even whether a particular widget will be a check box or a push button. As parts of the feature come online you - or your dev! - implement the corresponding part of your test libraries. The dev code and test code can be complete at virtually the same time and checked in to version control in the same checkin. Test can actually be in sync with Dev! And because Dev has been using the tests throughout, those tests are very likely to pass.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Let's say for the moment you buy my proposition that thinking in terms of user actions will help you create effective test automation libraries and test cases that are easy to write, easy to change as the product changes, and whose organization stays clear over time. How do you integrate that into your product's lifecycle? What does it look like?

I'll go into copious detail soonest, but here's an overview to whet your appetite in the meantime:

Every product development effort starts with a vision for what the product will be, why it's being built, what its point is. This vision isn't always clearly articulated, but I have yet to see a development effort started just because the team was bored and didn't have anything better to do. Test this vision. Do you even have one? Does everyone on your team know what it is, do they understand it? Is it coherent? Can you imagine the types of user actions it might entail?

Next comes an initial gathering of requirements. If you're working in an agile environment this may only last a day or two, with the user stories being recorded on note cards or a simple online list. If your team follows the classic waterfall process (poor you!) requirements gathering may last for months as the team attempts to etch in stone every last requirement your customers have. Regardless of where on this spectrum your team lives, be involved throughout, looking at the requirements through user action-colored glasses. Again, can you imagine the types of user actions each requirement might entail?

Posted by The Braidy Tester at 07:30 AM  Permalink |

For a while now I've been thinking that perhaps I have a book percolating in me. Maybe it's really multiple books - some of the topics seem to not exactly hang together. It occurred to me that blog posts might be a good way to write it: I get my ideas, theories, and opinions out of my head. You all read said content, have some reaction to it, and hurrah/disagree with/flame it by posting comments or emailing me directly. We have a lively discussion. Everybody learns a lot. Opinions are changed. The book (if such it turns out to be) gets way better. The acknowledgements paragraph goes on for pages and pages listing all y'all's names. <g/>

That's the plan, anyway. We'll see what happens. Here's what I have so far, an introductionry bit:

It's All About Your Customer: A Guide To Effective Test Automation

What's the point of testing? To provide information regarding how well your customers can use your product to get something done, to achieve some goal they have. I believe writing your test cases in terms of user actions leads directly to effective, useful test cases. This is so regardless of whether you are doing manual or automated testing, regardless of whether you are testing a user interface or a programmatic object model, regardless of which tool you happen to be using, regardless of which methodology you follow. Focusing on your customer will help you prioritize and direct your testing. It will help your test automation libraries stay consistently organized in the face of change. It will help you capture the essence of your test cases.

Posted by The Braidy Tester at 07:30 AM  Permalink |