THE BOOK OF TESTING

Thoughts From a Braidy Tester

September 2006

You are not done testing yet unless...you have vetted your application's readiness for use in international locales. Even if you are positive that your application will never be used anywhere other than your own country, these issues are worth at least investigating. Your team may decide not to fix problems you find, but at least y'all will know where they are. And, really, are you one hundred percent certain that someone from upper Elbonia *won't* want to use your product?

• Look for culture-specific images and terminology. For example, the color red is synonymous with danger in many Western cultures, but it symbolizes happiness or good luck in others.
• Look for geo-political sensitivity issues. One common example is maps that cover areas whose ownership or exact boundaries are disputed between countries. If your application contains or displays maps in any fashion, be prepared for all kinds of pain the moment anyone outside your country starts using them!
• Verify that your application correctly handles switching to different system locales, language packs, and code pages, both before your application has started and while it is running.
• Verify that your application correctly handles switching to different regional settings, both before your application has started and while it is running: date and time formats, currency symbols and formats, and sort orders, to name just a few. Some or all of these settings will vary across locales and language packs; most modern operating systems allow you to customize all of this separately from changing languages or locales as well. (On Microsoft Windows, do this via the Regional Settings control panel applet.) For example, if your application works with currency, see what happens when you change your currency symbol to "abc". One application I worked on did not like this at all!
• Verify that your application correctly handles multi-byte (e.g., Japanese), complex script (e.g., Arabic) and right-to-left (e.g., Hebrew) languages. Can you cursor around this text correctly? What happens if you mix right-to-left and left-to-right text?
• Verify that all controls correctly interact with Input Method Editors (IMEs). This is especially important if you intend to sell into East Asian countries.
• Verify that your application correctly handles different keyboard mappings. As with regional settings, certain locales and language packs will apply special keyboard mappings, but operating systems usually allow you to directly modify your keyboard map as well.
• Verify your application correctly handles ANSI, multi-byte, and Unicode text, extended characters, and non-standard characters on input, display, edit, and output.
• Verify that the correct sorting order is used. Sorting correctly is hard! Just ask anyone who has run into the infamous Turkish "i" sort order bug. If you rely on operating system-provided sort routines then you should be in good shape, but if your application does any custom sorting it probably does it wrong.
• Verify that the system, user, and invariant locales are used as appropriate: use the user locale when displaying data to the user; use the system locale when working with non-Unicode strings, and use the invariant locale when formatting data for storage.
• Verify that any language-dependent features work correctly.
• Verify that your test cases correctly take into account all of these issues. In my experience, testers make all the same mistakes in this area as do developers - and won't you be embarrassed if your developer logs a bug against your test case! <g/>

Posted by The Braidy Tester at 07:30 AM  Permalink |

Just like people in most other disciplines, testers often throw around terminology that other people might not understand. "Those bloody devs". <g/> Here's a decoder ring for some of those terms.

• BVT, or Build Verification Test. Also knows as a smoke test. A short, fast, and simple test meant to determine whether a daily build is so broken that it's not worth even installing, let alone testing.
• FVT, or Feature Verification Test. A BVT for a specific feature, meant to determine whether that feature is worth testing.
• Exit Scenario. Tests which must pass in order for the team to exit the current milestone. This set of tests will include customer-focused end-to-end scenarios. It may also include more narrowly scoped feature-centric tests.
• Basic Functionality. Also known as comprehensive tests. Any test that isn't a BVT, FVT, or Exit Scenario.
• Dailies, or Nightlies. The set of tests that is run on every daily (or nightly) build. Typically this will include at least BVTs and FVTs; often it includes exit scenarios as well.
• Positive Test. A test that covers expected use cases. Depending on your point of view, this class of tests might include common error paths.
• Negative. A test that covers unexpected use cases, such as entering crazy long text strings in a text box or attempting to use disabled functionality. Most error conditions are generally included here as well.
• User Scenario. A test case that reproduces a sequence of actions you expect your customer to take. User scenarios can be as narrow as "Select some text and make it bold", but they are typically more broad.
• Black Box. Testing based solely on what can be observed of an application's behavior.
• White Box. Testing based on knowledge of an application's implementation. Code inspections and code coverage analysis are two examples of white box testing.
• Grey Box. A mix of White Box and Black Box testing, where some knowledge of the application's implementation is used to guide observation-based testing.

Are there other terms you would like decoded? Let me know!

Posted by The Braidy Tester at 07:30 AM  Permalink |

James Whittaker is, I dare say, one of the celebrities of the testing world. He was long a professor of computer science at the Florida Institute of Technology, where he became well-known for his efforts to find ways to make testing a teachable skill. He and his research group there created innovative testing technologies and tools, including the popular runtime fault injection tool Holodeck, and became highly skilled at breaking software security. James founded Security Innovation to productize his work, but recently he has left both that company and teaching to join Microsoft as a Security Architect, where he is working to integrate testing into the Security Development Lifecycle (SDL).

James wrote How To Break Software - one of my favorite books on testing, co-wrote How To Break Software Security (also very good) with Hugh Thompson, and co-wrote How To Break Web Software (haven't read it yet) with Mike Andrews. James' talks at Microsoft are always standing room only; this interview will give you a taste of why.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless…you have searched out every alert, warning, and error message and dialog box your application can display and checked the following:

Content

• Verify that you understand every condition that can cause the alert to display, and that you have test cases for each condition (or have explicitly decided to *not* test specific conditions).


• Verify that the alert is in fact needed. For example, if the user can easily undo the action, asking them whether they really want to do it is not necessary.


• Verify that the alert first identifies the problem and then presents the solution. Basically, treat your customers like smart, knowledgeable people and help them understand what the problem is and what they can do about it.


• Verify that the alert text does not use an accusatory tone but rather is polite and helpful. I remember one application who scolded me with "You did not close me correctly" as it auto-recovered its database after crashing. Umm, no, *you* did not close yourself correctly; I had nothing to do with it! A later release of the application changed the message to "This application was not closed correctly", which is a little better. As was the case for me, almost certainly your customer did not cause the problem intentionally. If they did do it on purpose, likely they didn't know it would be a problem. Again, let them know what happened, what the application is doing to remedy the situation, and what they can do to prevent it from happening in the future.


• Verify the alert text is correct and appropriate for the situation.


• Verify the alert text is consistent in its wording and style, both to itself as well as to each other alert.


• Verify the alert text is as succinct as possible but no more succinct. Hint: If the alert text is longer than three lines, it's probably too long.


• Verify the alert text contains complete sentences which are properly capitalized and punctuated.


• Verify the alert text does not use abbreviations or acronyms. (Discipline-specific acronyms may be OK, if you are confident that all of your users will know what they mean.)


• Verify the alert text uses the product's name, not pronouns such as "we" or "I".

Functionality

• Verify the alert's title bar contains the name of the product (e.g., "Acme Word Processor").


• Verify each button works correctly. I have seen innumerable "Cancel" buttons that were really an "OK" button in disguise!


• Verify each button has a unique access key.


• Verify the buttons are centered below the message text.


• Verify any graphics on the alert are appropriate and correctly placed. For Microsoft Windows applications, there are standard icons for Informational, Warning, and Critical alerts, and these icons are typically displayed to the left of the alert text.

Posted by The Braidy Tester at 07:30 AM  Permalink |

When I was casting about to decide who to interview first, Gerald M. Weinberg (Jerry to everyone who knows him) seemed the obvious choice. Jerry hasn't been around forever, exactly, but he has been part of the computer world for a good many years. He worked for IBM as a computer - that is, a person paid to compute equations and such by hand. He has learned a lot about computers - and people - in the ensuing years, and much of what he has learned he has shared with the rest of us through a plethora of books (see the full list at the his website). He also shares his knowledge and wisdom via his Secrets of Consulting and Weinberg On Writing blogs, SHAPE forum, and a variety of conferences and workshops. Here's what he has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
GW: I'm an old, old timer. Back in 1956, we barely tested our programs. Instead, we made sure they were right the first time by careful analysis and technical reviews. (Our computer was 3,000 miles away by propeller plane, so mistakes were costly.) 90% of the time, our programs ran correctly the first time they touched the machine (not "a" machine, but "the" machine, because it was the only one of its kind in the world).

Our programs were pretty simple by today's standards (one of them was the world's first operating system), but in 1957, when I delivered a program with 3 wrong answers out of 20,000 results (as a result of a rare overflow condition), I started to be aware of the need for more careful examination of results, if not of explicit testing.

Then, in 1958, when we built the operating system that ran the space tracking network for Project Mercury, I became acutely aware for the need for testing because someone's life depended on our not delivering mistakes. Before that time, there were no "testers"--in fact, we weren't even called programmers yet. Everyone did their own testing within our team, and we did test-first development, though we didn't call it that. We established an explicit testing group--a bunch of "programmers" who were given the specific job of beating the system in any way possible, primarily by building simulation test drivers. We didn't kill any astronauts, so I guess we were successful, and the tradition continued through all subsequent space missions as well as in the newly created IBM Federal Systems Division, though there were still some holdouts that believed you wouldn't need testing if you were careful enough in doing your development. (I supposed that's still true--but who is ever careful enough?)

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
GW: The biggest surprise is how many people still believe you can test quality into a program.

DDJ: What is the most interesting bug you have seen?
GW: Oh, that would be hard to say. I guess the one I'm currently faced with is always the most interesting. I've written somewhere about a bug in that Mercury Project that was very difficult to find, and would have killed people if we hadn't persisted in tracking it down--in spite of management trying to write it off as a random coincident machine failure (which had about one chance in 10**100000000000 of being true).

DDJ: How would you describe your testing philosophy?
GW: I don't have one. It's not about philosophy. (Or even less, methodology.) Philosophy is the love of knowledge. Methodology is the study of method. We've corrupted the words.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
GW: To know: That they're not perfect, just like everybody else.
To do: Be humble.
For developers:
To know: That they're not perfect, just like everybody else.
To do: Be humble.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
GW: Pounding mindlessly on keyboards.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
GW: To get management to understand what testing can and cannot do.

DDJ: Going meta (to steal your favorite technique), what else should I ask you? What would you answer?
GW: Q: "How can I learn to think better?"
A: (of course) Read my books, subscribe to my SHAPE Forum, and participate in the AYE Conference.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Today I'm excited to announce that, well, I have this crazy idea that if I email random people asking them questions about testing they will respond. <g/> For the interview, I'll provide the following list and ask them to pick five to answer.

• What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
• What has most surprised you as you have learned about testing/in your experiences with testing?
• What is the most interesting bug you have seen?

• How would you describe your testing philosophy?
• What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
• Is there something which is typically emphasized as important regarding testing that you think can be ignored, that is unimportant?

• What do you see as the biggest challenge for testers/the test discipline for the next five years?
• Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
• Is there anything else you would like to say?

"That's quite a lot of questions!" you may be saying. Indeed. That is one reason why I am asking my interviewees to answer just five of these questions - their choice which five.

I will update this post as I publish people's responses, making it a sort of table of contents. First up, on Monday, is Jerry Weinberg!

If you have a suggestion for someone to interview, let me know! Here's who I've interviewed so far:

• Scott W. Ambler


• Anu Arora


• James Bach


• Jon Bach


• Kent Beck


• Kevin Beto


• Rex Black


• Bliss


• Michael Bolton


• Lyne Brown


• Bill Buxton


• Dave Catlett


• Paul Carvalho


• John N. D’Addamio


• Selena Delesie


• Bob Dewing


• Elfriede Dustin


• Danny R. Faught


• Sara Ford


• Tom Gallagher


• Evan Goldring


• Adam Goucher


• James Hamilton


• Elisabeth Hendrickson


• Matthew Heusser


• Douglas Hoffman


• Eric Jacobson


• Jonathan Kohl


• Shrini Kulkarni


• Brian Marick


• Chris McMahon


• Joe McMahon


• Wayne Miller


• Jim Moore


• Robert Musson


• Alan Myrvold


• Alan Page


• Alvaro Peon-Sanchez


• Jared Quinert


• J. B. Rainsberger


• James Rodrigues


• Bj Rollison


• Johanna Rothman


• Eric Sink


• Ross Smith


• S. Somasegar


• Pradeep Soundararajan


• Robert Straavaldson


• David Treadwell


• Gerald M. Weinberg


• Adam White


• James Whittaker


• Joshua Williams


• Mike Zintel

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have checked how your application handles printing. If you remember back to the Bad Old Days before your operating system abstracted away (most of) the differences between printers, so that each application had to know intimate details about every printer it might be used with, you surely know how good you have it. That just gives you more time to worry about the following issues.

• Verify changing orientation works properly. Try doing this for a brand new document and for an in-progress document. Also try doing this by launching your app's equivalent of a page setup dialog box both directly (e.g., from a menu item) and from within the print dialog.
• Verify printing to a local printer works properly.
• Verify printing to a network printer works properly.
• Verify printing to a file works properly. Every operating system I know of allows you to create a print file for any printer you have installed.
• Verify printing to a PCL printer works properly. PCL started out as the control language for Hewlett-Packard printers but has since become somewhat of a standard.
• Verify printing to a PostScript printer works properly. This printer control language was created by Adobe and has also become somewhat of a standard. PostScript is semi-human readable, so you can do some printer testing by inspecting the output file and thus avoid killing any trees.
• Verify printing to a PDF file works properly. There are a number of free and low-cost PDF creators available; also consider purchasing a copy of Adobe Acrobat in order to test the "official" way to create PDFs.
• Verify canceling an in-progress print job works properly. My current printer pretends to let me cancel, but then pages come spitting out anyway. Frustrating!
• Verify setting each print option your application supports has the proper effect; number of copies, collation, and page numbering, for example.
• Verify setting printer-specific options works properly. These settings should be orthogonal to your application's print settings, but you never know.

Posted by The Braidy Tester at 07:30 AM  Permalink |

A while ago I posed a few puzzles I stole from Jerry Weinberg's An Introduction to General Systems Thinking. The first was to see how many different plausible next members you could think of for the set

{ 1, 2, 3, ... }

It seems there should be tons, but I only came up with the following:

• The set of integers, making the sequence { 1, 2, 3, 4, 5, 6, 7, ... }
• The set of prime numbers, making the sequence { 1, 2, 3, 5, 7, 9, 11, 13, 17, ... }
• The Fibonacci Sequence, making the sequence { 1, 2, 3, 5, 8, 13, 21, 34, ... }
• A pattern of odd-even-odd, making the sequence { 1, 2, 3, 5, 6, 7, 9, 10, 11, ... }
• A sequence of random numbers that just happens to start with three numbers in order, giving an infinite variety of sequences

The second puzzle was to see how many different plausible next members you could think of for the set

{ Mathew, Mark, ... }

Reader Rob B. came up with

• [ The set of all words ]
• [ The set of all words that begin with "M" ]
• [ The set of all words that are either four or seven characters in length ]
• [ The set of all words that use at least one letter from each of the three rows on a Qwerty keyboard ]
• [ The set of all words made up only from the letters A, E, H, K, M, R, T, W ]
• [ The set of all words that do not include the letter B (or the letter C, or... There are nineteen of these) ]
• [ The set of all words that include neither the letter B nor the letter C (or neither B nor D, or … ) ]
• [ Lots of variants on words that do not include a specific sequence of letters ]
• [ Repeat all these sequences for "the set of all names" rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all male names" rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all Christian disciples " rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all Christian apostles" rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all names from the Bible" rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all male names from the Bible" rather than "the set of all words" ]
• [ Repeat all these sequences for "the set of all names for and types of currency" rather than "the set of all words" ]

• The gospels in the Bible, making the sequence { Mathew, Mark, Luke, John }
• People who have ever been members of the United States Congress, making the sequence { Mathew, Mark, Maxine, James, Louise, ... }
• [ Lots of variants on governing bodies ]
• People who are dead
• People who are alive
• People who were born overseas but are American citizens by birth
• [ Lots of variants on groups of people ]

So how does this apply to testing? The answer is yet another set of probably infinite size.

One way it applies is as a reminder that there are always several different solutions to every problem. Keep this in mind as you ponder how best to test your feature. For example, find out how your developer implemented your feature, and then consider how else they might have implemented it. Think about how each of those implementation options is likely to break, and then try breaking your feature in each of those ways.

Another way this applies to testing is as a reminder that other people think differently than you. Rob and I came up with several of the same possibilities for the second set above, but we also came up with several different answers as well. If you run your planned test cases past the other testers on your team, and your devs, and whomever else you can get to listen, not only will they have a better idea of what you are doing, but almost certainly they will suggest tests that you did not think of. That certainly can't hurt the quality of your product!

How else? Let me know.

Posted by The Braidy Tester at 07:30 AM  Permalink |

There's this one parking lot I walk through on my way to work each day. One morning I discovered that the blacktop elves had been hard at work the night before resurfacing the lot. They weren't able to finish, however, so I could tell that they simply laid a very thin layer of new blacktop on top of the existing surface. They finished the next night, and now the parking lot looked brand new.

Well, it did for a day or two anyway. Then the pebbles and rocks embedded in the tar started popping out, leaving holes behind. Now it's not in much better shape than it was before the resurfacing.

In a previous job we called this sort of thing "putting lipstick on a pig". We weren't talking about resurfacing parking lots but rather the unfortunately common practice some companies seem to have of slathering ritzy new features on top of a creaky foundation in the hopes that users won't realize that none of the important things have changed. In the case of the parking lot, it sure looks all shiny and new, but a closer inspection shows that the shininess doesn't really even go skin deep.

One of my guiding mantras is to always pursue quality, not quantity or speed. I find that going a bit more slowly and taking the time to do things right ultimately lets me go faster and accomplish more. Stripping and completely resurfacing the parking lot certainly would have taken longer than just reoiling it did, but it also would have lasted much longer.

Quality really is Job Number One!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have tested your application in the following special modes and states. Ideally you would run each of your tests in each of these special cases, but I haven't yet met anyone who has that much time. More typical is to pick one case each day as the context in which to run your tests that day.

• Different zoom levels, as appropriate. One application I know of had a huge number of automated tests, and everything seemed hunky dory. Then someone thought to try changing the zoom level and found all sorts of issues. Oops.
• Safe Mode. Microsoft Windows has a special mode where just the essentials are loaded - the most basic display driver, a bare-bones network stack, and no start-on-boot services or applications. How does your app handle being run under these conditions?
• Sharing documents between multiple users and/or multiple machines, simultaneously and sequentially. This is especially important for programs that access a database (what happens when someone else makes a change to the record you are editing?), but if you can open documents off a network share, or you can open documents from a shared location on the local machine, someone else can do so as well - potentially the very same document you are editing.
• No file open, dirty file open, dirty-but-auto-saved file open, saved file open.
• Full screen and other view modes.
• Different application window sizes (document window sizes too, if your app has a multi-document interface); especially: default launch size, minimized, maximized, not-maximized-but-sized-to-fill-the-screen, and sized very small.
• Invoke standby, hibernation, and other power-saving modes whilst an operation is in progress. I discovered the hard way that when one application has a modal dialog up it blocks my OS from sleeping. I was *not* a happy camper!
• Resume your computer out of various sleep modes. Do in-progress operations continue where they stopped? Or do they restart? Or do they hang?
• Modified system settings. Set your mouse to move faster or slower. Change your keystroke repeat duration. Mess with your system colors. Does your application pick up the new values when it starts? Does it pick up values that change while it's running?
• Object Linking and Embedding (OLE). If your app supports OLE, you're in for a world of painfun! Does embedding other OLE objects in your app's documents work correctly? What about embedding your app's documents in other OLE-enabled applications? Do embedded applications activate and deactivate correctly? Do linked OLE documents update when the source of the link is modified? How does your app handle the linked document's application not being available?
• Multiple selection. What happens if you apply text formatting when you have three different text ranges selected? Or you paste when several different items are selected? What should happen?

The last two special states are not contexts in which to execute your test cases but rather additional tests to run at the end of each of your test cases:

• Send To. Many applications today have a handy menu item that lets you send the current document to someone as an email. I've seen nasty bugs where attempting to use this handy menu item crashes the application or worse.
• Cut, copy, and delete. To and from the same document, a different document, competing applications, targets that support a less-rich or more-rich version of the data (e.g., copying from a word processor and pasting into a text editor), targets that don't support any version of the data (what happens if you copy from your file explorer and paste into your application?)...you get my drift.

Posted by The Braidy Tester at 07:30 AM  Permalink |

One of my colleagues recently told me about his first day on his first job as a tester. Once he was settled in his manager pointed him at one of the team's applications and said "This app has a nasty bug. Find it."

Five minutes later my colleague was in his boss's office. "I found the bug!"

His manager shook his head. "That's a good bug, but not the one I was talking about."

Five minutes later this scene repeated itself. Over and over and over that day, my coworker asked "Is this the bug?" and his manager replied "Nope."

By the end of the day - his first day ever as a tester - my coworker had found some eighty high-quality bugs. None of them were the one his manager wanted him to find - if, in fact, his boss ever even had a specific bug in mind. But eighty bugs in one day! That's doing pretty good!

I see several morals in this story:

• Bugs are easier to find when you know they are there. And as every great tester knows, there are always more bugs.


• Different people react in different ways to the same situation. Had I been in my colleague's place, I would have rapidly become enormously frustrated. He, however, became ever more highly motivated to find that one specific bug his boss had him searching for. Understanding what makes your teammates tick can greatly improve your relationships with them.


• Testing is fun! As my coworker recounted this story, I could tell that it is a happy memory for him. The best testers I know all have stories about gnarly bugs and flaming apps and working with developers of various degrees of tester-friendliness which they recount with glee and enthusiasm at the drop of a hat. And isn't having fun the point of life?

Thanks Dante for this story! What are yours?

Posted by The Braidy Tester at 07:30 AM  Permalink |