THE BOOK OF TESTING

Thoughts From a Braidy Tester

July 2006

As I was walking down the street the other day I noticed a sign in the window of a restaurant:

Driver Wanted

Valid WSDL required

I blinked, and then blinked again. WSDL required? The restaurant needing a delivery driver made sense, but why would the driver need WSDL? Does it matter what sort of API that WSDL defines, or will any old WSDL do? Is the owner one of those executives who gets everything they know from airline magazines, and who has been seeing this WSDL thing show up in so many articles that he figures it's time for him to get one? Are they going to embed a wireless-enabled chip in the driver so they can download new instructions and driving directions directly into the driver's brain? How do they check quality of service? Have the driver go all over town and ask "Can you ping me now? Good. Can you ping me now? Good."

Eventually, of course, I realized that WSDL stood for Washington State Driver's License, and suddenly everything made sense. The person who made the sign likely never even considered that it could be interpreted differently than they intended it.

If the sign was intended to be immediately understood by all comers then it had a bug. The person who made the sign didn't catch the bug because they didn't move out of their frame of reference. Or perhaps they did, but decided the bug wasn't a stop-ship problem. Either way, I ran into it. There was nothing in my path to stumble over as I did a double-take looking at the sign, so I wasn't hurt. But of course bugs are not always so benign.

Everybody knows software has bugs, but everything else in life has bugs as well. Don't limit your testing to your application, but rather test everything you come into contact with. But do be careful not to go too far - most people don't like it when you test their artwork!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless…you have checked the following points for each and every dialog box in your application:

• Verify that each command (e.g., menu item, shortcut key) that is supposed to launch the dialog box does in fact launch it.
• Verify that its title is correct.
• Verify that all terms used by the dialog are consistent with those used by the rest of the application.
• Verify that accepting the dialog box updates application state correctly.
• Verify that canceling the dialog box causes no change to application state.
• Verify that the dialog is sticky - displays in the position from which it was last dismissed. Or that it always displays in the same location, if it is not supposed to be sticky.
• Verify that the dialog's contents are initialized from the current state of the application. Or that it always starts with default values, if it is not supposed to initialize from application state.
• Verify that invoking help (e.g., pressing F1) links to the correct help topic. Note that you may need to check this for each individual control as some dialog boxes have control-specific context-sensitive help.

Posted by The Braidy Tester at 07:30 AM  Permalink |

"Very nice equivalence classing, Bianca!" Daphne said.

"Off the page is an important test. We'll want to check all four types of off the page - too far left and right horizontally, and too far up and down vertically."

"Hey, what happens if the computer's screen resolution is too low to show the entire drawing?" asked Lucas. "And what happens if the screen resolution changes while this Rubbings app is running?"

"Oh, we can just do a standard scrolling canvas" replied Jason.

"But are children going to understand how to deal with that?" queried Bianca. "I'm not sure my two-year-old niece would. Maybe we should just auto-zoom the page instead."

"Oliver, can you look into that?" Hazim asked. Lucas added "Different screen resolutions" and "Changes in screen resolution" to the list.

"Let's get back to the drawing tools" Daphne said. "Bianca's equivalence classes were right on. We should do exactly at the corner and one pixel off from each corner."

"So for example, 0,0; 1,0; 1,1; 0,1; -1,1; -1,0; -1,-1; 0,-1; 1,-1?" checked Lucas.

"Exactly!" said Jason. Lucas added those cases to the list.

"And just like Bianca said", continued Jason, "we'll want to do the four quadrants and smack dab in the middle of the page too."

"For fill, the color being filled seems like it could matter" Daphne said. "We should try filling a different color from what we're filling with, and the same color. And something that has never been filled - the page's background."

"And we should see what happens if we fill an area defined by each of the tools" added Bianca. She was really getting into the spirit of things. "Theoretically it shouldn't matter, but still..."

"Very nice!" Jason said approvingly. "We'll make a tester out of you yet!" he grinned.

"For the stamps we have another cross-product matrix" Daphne said, "switching to every stamp from every other stamp. And to every tool from every other tool, for that matter. And we should make sure that each stamp actually stamps the right shape."

"This is a great set of test cases!" Hazim said. "And clearly you all really could continue coming up with more and more and more. If we think of more later we can easily add them, right?"

"Yesireebob!" the team chorused.

"So let's stop here and prioritize the test cases we have so far" Hazim suggested.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Today's link comes from Developer.*, where Scott Sehlhorst suggests we Test Smarter, Not Harder. Scott covers random sampling, pairwise and n-wise testing, and the (sometimes un)importance of operation order. Scott does an excellent job of explaining these testing techniques in terms that even your manager can understand! Along the way he talks about the difference between your confidence in an application and your app's quality, and also equivalence classes. And then he sums it all up in a few bullet points ripe for rapid review. Give it a read today!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have verified your application plays nicely with the accessibility features of your operating system. Accessibility features are vital to customers who are blind, deaf, or use assistive input devices, but they are also extremely useful to many other people as well. For example, comprehensive large font support will be much appreciated by people with failing eyesight and/or high DPI screens.

Some of the following terms and utilities are specific to Microsoft Windows, but other operating systems likely have something similar.

• Verify that every control on every dialog and other user interface widget supports at least the following Microsoft Active Accessibility (MSAA) properties:
  • Name - its identifier
  • Role - a description of what the widget does, e.g., is it invokable, does it take a value
  • State - a description of its current status
  • Value - a textual representation of its current value
  • KeyboardShortcut - the key combination that can be used to set focus to that control
  • DefaultAction - a description of what will happen if the user invokes the control; e.g., a checked check box would have a Default Action of "Uncheck", and a button would have a Default Action of "Press"
• Verify that changing the value of each control updates its MSAA State and Value properties.
• Run in high contrast mode, where rather than a full color palette you have only a very few colors. Is your application still functional? Are all status flags and other UI widgets visible? Are your toolbars and other UI still legible? Does any part of your UI not honor this mode?
• Run in large font mode, where the system fonts are all extra large. Verify that your menus, dialogs, and other widgets all respect this mode, and are still legible. Especially be on guard for text that is truncated horizontally or vertically. To really stress your UI, do this on a pseudo-localized build!
• Run with Sound Sentry, which displays a message box, flashes the screen, or otherwise notifies the user anytime an application plays a sound. Verify that any alert or other sound your application may play activates Sound Sentry.
• Run with sticky keys, which enables the user to press key chords in sequence rather than all at once. The operating system will hide much of these details from your application, but if your app ever directly inspects key state it may need to explicitly handle this state.
• Run with mouse keys, which enables the user to control the mouse pointer and buttons via the numeric keypad. Again, the operating system will hide much of these details from your application, but if your app ever directly inspects mouse state it may need to explicitly handle this state.
• Run with no mouse and verify that every last bit of your UI can be accessed and interacted with solely through the keyboard. Any test case you can execute with a mouse should be executable in this mode as well.
• Run with a text reader on and your monitor turned off. Again, you should be able to execute each of your test cases in this state.
• Verify focus events are sent when each control loses and receives focus.
• Verify the tabbing order for each dialog and other tab-navigable UI component is sensible.
• Verify that any actionable color item (e.g., that red squiggly line Microsoft Word displays underneath misspelled words) can have its color customized.
• Verify that any object which flashes does so to the system cursor blink rate.

How completely you support these various accessibility features is of course a business decision your team must make. Drawing programs and other applications which incorporate graphics, for example, may decide to require a mouse for the drawing bits. As is also the case with testability, however, accessibility-specific features are often useful in other scenarios as well. (The ability to use the keyboard to nudge objects in drawing programs tends to be popular with customers of all abilities, for example.)

Posted by The Braidy Tester at 07:30 AM  Permalink |

"Oh that's stupid!" Bianca complained. "Kids aren't going to be going around making drawings read-only!"

"Remember the ground rules!" Hazim interjected. "We aren't judging right now."

"Yeah yeah yeah" Bianca grumbled. She still didn't seem happy about the read-only test case, but she didn't keep up her fight.

"Saving may be automatic", Lucas said, "but we still need to save to a variety of locations: hard drive, floppy drive, removable media like a Zip drive."

"USB drive", Bianca continued, "CD-ROM (which I guess would be read-only, my apologies oh mighty testers)."

"CD-RW, DVD-ROM, DVD-RW, network share." Bianca and Lucas finished together in rather a sing-song voice. Clearly they knew this list well.

"I guess developers really can learn testing checklists!" Daphne smirked to Jason.

"Um, if saving is automatic, how do we save to all those different locations?" Oliver queried.

"That is something for our fearless leader Hazim The Not Quite Handsome to determine!" Bianca replied jokingly.

"Printing is going to be simply loads of fun" Lucas said sarcastically. "Windoze may provide standard printer APIs but printers sure don't react to them in standard ways!"

Lucas added "Print to loads of different printers" to the list of brainstormed test cases. "I don't remember whether we need to deal with all those printer options" he said, "or whether the printer drivers lay them on top of whatever data we feed them."

"Probably we want to hide those anyway" Oliver said. "They aren't exactly kid-friendly."

"They aren't exactly adult-friendly either!" exclaimed Daphne. "I never can find that blasted staple option."

"So what's left?" asked Bianca. "Oh yeah, coloring in the drawing."

"We said we would have a preset selection of colors, right?" asked Jason. "So we should make sure we can actually select each of those colors."

"Not just select each of them, but switch from every color to every other color" added Daphne. "That's a straightforward cross-product matrix."

"The tools all seem very similar" Bianca said. "We will want to try using each of them in the four corners of the page, in the middle of the page, and off the page."

"Very nice equivalence classing, Bianca!" Daphne said.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Bianca brightened. "Now that's a plan I can be happy with" she said. "Let's invent us some test cases!

Running a brainstorming session was something Hazim didn't have to learn. "Remember the brainstorming ground rules: No idea is stupid, so don't censor one even if you think it's not any good. We'll cull through everything after we're done, but for now we're going for quantity more than quality.

"I'll start: Probably we should open each and every coloring book, and then select each and every page in each book, just to be sure we can."

"Check!" Lucas was serving as scribe. "Hey, do we have to browse one-by-one through each book on the shelf, and then through each page in the book, or can we jump right to the book or page we want?"

"Probably we'll want to do both" Hazim answered.

"For jumping", Oliver said, "we could do just like you would with a real book: show an angle-on view of the edge of the book and let the user indicate to where they wish to open the book. Then from there they can browse to the exact page they want."

"We had separate Choose Page and Open actions" Daphne said, "but I'm thinking now that they're really the same - Choose Page simply starts a new drawing while Open opens an existing drawing."

"One difference is that Choose Page copies the page", replied Bianca, "while Open opens the drawing. That seems like two different actions to me. Choose Page is for starting a new drawing based on an existing one, whereas Open is for continuing work on an existing drawing."

Oliver looked thoughtful. "What if we always start a new drawing? This would make it easy for a rug rat to go back to a previous version, even if they hadn't specifically decided to allow for that possibility. If the child changes the drawing's name, that would effectively be a Save As. If they don't change the drawing's name, we could simply keep a timestamped history of the drawing."

"I like it!" Lucas exclaimed. "And Save should be automatic, just like in Microsoft OneNote. You don't have to explicitly save paper drawings, after all."

"I'll have to do some thinking about that history UI" Oliver said. "But I like it too."

Hazim smiled a big smile. He always loved to see a team firing on all cylinders without evident need of his help. Invisible leadership was more his style, and he was glad to see that that seemed to be what this team would need. "I'll check with Marketing, but those sound like great ideas" he said.

"So naming the drawing seems even more important now" Bianca said. She pointed towards Jason and Daphne. "I remember you two have a crazy number of string and filename test cases."

"Yup" replied Daphne. "We'll want to go through our standard set of string and filename test cases."

"We should check what happens if you name a drawing the same as another drawing" added Hazim. "Both the same name as a drawing in that same coloring book and as a drawing in some other coloring book."

"Oh - and we should try naming a drawing the same as another drawing that is read-only!" Daphne said. She and Jason grinned evilly at each other - they have found read-only files to be a common trouble spot.

"Oh that's stupid!" Bianca said. "Kids aren't going to be going around making drawings read-only!"

Posted by The Braidy Tester at 07:30 AM  Permalink |

DDJ selected me as one of the developers they diaried in their August issue. Even though I'm not a developer! Read more at http://www.ddj.com/189500004?pgno=3.

Posted by The Braidy Tester at 07:30 AM  Permalink |

"We have a start on more test cases" Jason replied, shooting dark looks at Bianca. "But we'd rather have you all brainstorm them with us rather than doing it all ourselves."

"Oh geez this is going to take all day!" Lucas grimaced. "You two always dream up huge numbers of test cases!"

"That's why we're the testers!" Daphne teased.

"Not that customers would ever do half of them" Bianca groused.

"Now that's not true" Oliver replied. "Remember how you said that last time, and then the first thing people did in the usability studies was exactly that sequence of steps you said they would never do?"

"Well, ..." Bianca still looked petulant.

"Besides, it doesn't matter whether customers will do them or not" Hazim stepped in. "I haven't known you long, Bianca, but from what I can tell so far you don't intentionally write bugs, do you?"

"Of course not!" Bianca replied indignantly.

"So if a test case catches a bug that sneaks in, that's a good thing? Even if a customer never would have run into it?" Hazim continued.

"I suppose." Bianca seemed to be warming up to the idea, but she clearly had a ways to go.

Jason had been trying to get in to the conversation and finally succeeded. "You do have some good points, Bianca. Our devious minds never stop thinking of potential problems, but just because we can think of something that might go wrong doesn't mean we necessarily care whether it does. One change we would like to make to our process this time around is to have the entire team prioritize test cases, just like we prioritize features. That way we always have the most important tests finished, just like we always have the most important features finished."

Bianca brightened. "Now that's a plan I can be happy with" she said. "Let's invent us some test cases!

Posted by The Braidy Tester at 07:30 AM  Permalink |

"This is great!" Hazim said. "Needing to name drawings was a good catch. I'll run this feature set past Marketing and see if they agree. Wait a minute - you said you have test cases already?"

"Well, a first draft of the Build Verification Test [BVT], at least" replied Daphne. "Here's what we're thinking." She switched the wall display to her code editor.

Logical.Application.Start()

Logical.ColoringBooks.OpenColoringBook("BasicBook")

Logical.ColoringBooks.ChoosePage(23)

Point topLeft = new Point(42, 24)

Point topRight = new Point(350, 50)

Point bottomRight = new Point(320,300)

Point bottomLeft = new Point(70,180)

Logical.Color.DrawLine(topLeft, topRight)

Logical.Color.DrawLine(topRight, bottomRight)

Logical.Color.DrawLine(bottomRight, bottomLeft)

Logical.Color.DrawLine(topLeft, bottomLeft)

Point insideShape = new Point(150, 150)

Point outsideShape = new Point(10, 10)

Logical.Color.FillArea(insideShape)

Logical.Color.UseColor(System.Color.Magenta)

Logical.Color.FillArea(outsideShape)

Logical.Color.UseColor(System.Color.NavyBlue)

for (int x = 25 to 400)

    for (int y = 25 to 400)

        Logical.Color.Stamp(Logical.Shape.FivePointedStar, new Point(x, y))

Logical.Application.Close()

"I can understand that!" Hazim exclaimed. "Is this an actual test case?"

"Yep!" Jason replied. "We simply write a function for each user action, and then write our test cases in terms of those actions. Test cases stay simple and easy to understand - their essence comes shining through!"

"But what about verification?" Hazim asked. "I guess the user action functions verify themselves?"

Lucas jumped in. "Yes, kind of. It's actually off in a separate verification model that isn't really connected to the user action functions - which, by the way, we call the Logical Functional Model [LFM] - but for now you can think of the verification as being inline to those LFM methods."

"How do you know which different way to perform an action to use?" queried Hazim. "Although I guess we only have one way to do each action in this application..."

"You're right that it seems that we don't have multiple implementations for any of our user actions right now", Daphne responded. "But even if we did, generally we don't want test cases to know that any options exist."

"Which caused a lot of discussion when we made that decision" interjected Bianca. "You might even say arguments. Especially for our BVTs - certain people were of the opinion that smoke tests should always do exactly the same thing, every time."

"Certain people still think that", muttered Lucas, with decided head-nodding from Oliver and Jason.

"But eventually we all agreed that a build is OK as long as one path works" Bianca continued.

Hazim looked troubled. "But what if the path that is picked isn't the one that works? What do you do then?"

Jason and Oliver pounced. "That's exactly what we still want to know!" they chorused.

"Well let's not get into that now" Bianca hurried along. "What other test cases did you two come up with?"

"We have a start on more test cases" Jason replied, shooting dark looks at Bianca. "But we'd rather have you all brainstorm them with us rather than doing it all ourselves."

Posted by The Braidy Tester at 07:30 AM  Permalink |

Daphne and Jason had been whispering and writing off to the side while Hazim and Lucas were talking. "Here's our test API for R3" Jason said. Daphne activated the new page she and Jason had added on her tablet.

• Logical.ColoringBooks.OpenColoringBook(string name)
• Logical.ColoringBooks.ChoosePage(int id)

• Logical.Drawing.Name(string name)
• Logical.Drawing.Open(string name)
• Logical.Drawing.Save()
• Logical.Drawing.Print()

• Logical.Color.UseColor(Color color)
• Logical.Color.FillArea(Point location)
• Logical.Color.DrawLine(Point start, Point end)
• Logical.Color.Stamp(Shape shape, Point location)

"Browsing the coloring book library feels like something our customers would do along the way to opening a coloring book, not an action all by itself" Daphne started out. "We assumed that coloring books will have a name; our users may or may not see these names in our UI, but our tests need some way to identify coloring books."

"Thinking about coloring book names we realized that each drawing will need a name as well" Jason continued. "Kids may select drawings by looking at thumbnails, but some children - older ones especially - likely will want to name their drawings. So we added a name method. Opening a drawing uses that name. And then of course drawings can be saved and printed."

"Why doesn't Save take the name to save the drawing as?" asked Bianca. "Or I guess that would be Save As maybe - do we need one of those?"

Daphne took that one. "We think that Save As goes beyond what most young children will understand. Maybe existing drawings should appear in a special coloring book that lets them make a copy of a drawing. That would fit with the blank page strategy Hazim came up with. As for save, if the drawing hasn't been named yet we can give it some default name. As with the coloring book name it might not be something the user sees, but our tests will need one to be there.

Jason finished up. "For coloring in the current page, we figure there will be a hard-coded set of colors for the rug rats to choose from - like the twelve colors you get in the watercolor boxes at the grocery store. The other drawing methods all use whatever the current color is. We thought about adding a DrawArc method, but we thought that might be too complicated for our younger customers."

"Why isn't there a method to select which stamp to use?" Hazim asked. "Oh - that's just an incidental detail, isn't it?"

"Yep!" Oliver jumped in. "Most kids aren't going to get all excited about spending time selecting shapes - they are going just going to say 'Oh, I want to stamp out a rectangle now.'"

"The 'Drawing' and 'Color' area names feel a little weird to me" Oliver said.

"Yeah, to us too" Daphne replied. "But we couldn't think of better names. 'Drawing' is overloaded between the thing you're making and the action of making it. And 'Color' could mean both the color that will be used when something is drawn and the act of coloring the drawing. We decided to go with 'Drawing' as the thing we're making and 'Color" as the action of making it. But we're not attached to those names by any means; other suggestions are very welcome."

"We came up with an initial set of test cases, too" Jason said.

"This is great!" Hazim said. "Needing to name drawings was a good catch. I'll run this feature set past Marketing and see if they agree. Wait a minute - you said you have test cases already?"

Posted by The Braidy Tester at 07:30 AM  Permalink |