THE BOOK OF TESTING

Thoughts From a Braidy Tester

May 2006

Unable to find a solution for that hard problem you have to solve? Be it how repro a bug, how to test that seemingly untestable feature, or how to convince your boss you need a raise, Michael Michalko's books Cracking Creativity and Thinker Toys are jam-packed with great ideas for getting your creative juices flowing. Pick a "game" at random, give it a go, and see what happens!

Posted by The Braidy Tester at 07:30 AM  Permalink |

Last week's quiz was testing the instructions for connecting to my hotel's LAN:

High Speed Internet Access

For support contact the Millennium Technology Help Desk, extension 9999, option 2

Four easy steps to connect:
1) You need a network interface card (NIC) with an adapter OR and [sic] internal NIC (Ethernet port).
2) Shut down your computer. Connect the high speed internet access cable on the desk to the adapter or Ethernet port.
3) Turn on your computer and launch an Internet browser.
4) The hotel welcome page will appear. Follow instructions to complete the connection.

Here are some questions I had:

Hotel security: Please note that I didn't actually try (most) of these, most especially that last one. <g/>

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...
You have tested the following boundary conditions for filenames:

• Single character filenames
• Short filenames
• Long filenames
• Extra-long filenames
• Filenames using text test cases
• Filenames containing reserved words
• Just the filename (file.ext)
• The complete path to the file (c:\My\Directory\Structure\file.ext)
• A relative path into a subfolder (Sub\Folder\file.ext)
• A relative path into the current folder (.\file.ext)
• A relative path into a parent folder (..\Parent\file.ext)
• A deeply nested path (Some\Very\Very\Very\Very\Very\Deeply\Nested\File\That\You\Will\Never\Find\Again\file.ext)

Filenames are interesting creatures and a common source of bugs. Microsoft Windows applications that don't guard against reserved words set themselves up for a DOS attack. Applications on any operating system that allow any old file to be opened/saved/modified leave a gaping hole onto "secured" files. Some users stuff every document they've ever created into their user folder. Other users create a unique folder for each document. Certain characters are allowed in filenames that aren't allowed elsewhere, and vice versa. Spending some focused time in this area will be well worth your while.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Today's link is How to Break Software, by James A. Whittaker.

James packed his book to the gills with easy-to-understand and easy-to-apply techniques that will almost certainly bring your application down in flames. Even if you're an experienced tester, you should read this book. If you're a newbie tester, this book will be invaluable. And if you're a developer, this book will give you insight into how your tester thinks.

If you developers go a step further and apply these techniques to your code *before* passing that code off to your tester, and then fix the bugs you find, not only will you make your tester happy but you will free them to find the really gnarly bugs!

Posted by The Braidy Tester at 07:30 AM  Permalink |

I'm in Orlando this week speaking at STAR East. The elevators in my hotel have flat screen monitors embedded in the walls. What's the first thing I did? Poke them to find out whether they were touch screens of course! My wife was aghast. <g/>

Know a good story? Email me and I'll highlight it here!

Posted by The Braidy Tester at 07:30 AM  Permalink |

I was recently in a hotel that provided the following instructions for getting Internet access:

High Speed Internet Access

For support contact the Millennium Technology Help Desk, extension 9999, option 2

Four easy steps to connect:

1. You need a network interface card (NIC) with an adapter OR and [sic] internal NIC (Ethernet port).


2. Shut down your computer. Connect the high speed internet access cable on the desk to the adapter or Ethernet port.


3. Turn on your computer and launch an Internet browser.


4. The hotel welcome page will appear. Follow instructions to complete the connection.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...

You have covered the following boundary conditions for every text entry field in your application. (Don't forget about editable combo boxes!)

• Zero characters


• One character


• Two characters


• Some characters


• Many characters


• One less than the maximum allowed number of characters


• The maximum allowed number of characters


• One more than the maximum allowed number of characters


• Spaces in the text


• Symbols (e.g., colon, underscore) in the text


• Punctuation in the text


• ASCII characters


• High ASCII characters


• German characters


• Japanese characters


• Hebrew characters


• Arabic characters


• Unicode characters from multiple character ranges


• Control characters

Posted by The Braidy Tester at 07:30 AM  Permalink |

Last week's quiz was testing the "select a font" dialog from your favorite word processor.

Tony submits that "since the font is mandatory while the others such as style and size are options [the answer is] (number of fonts) x (number of font styles + 1) x (number of font sizes + 1) x (number of effects + 1). So for example if there are 5 fonts, 3 styles (regular, bold, italic), 5 sizes and 2 effects (strike through, underline) then the number of test cases is 5 x 4 x 6 x 3 = 360."

Tony is close. He omitted combinations of font styles (that is, you can have both bold *and* italic applied) as well as combinations of effects. Missing also is any mention of selecting the font and font size via the list versus by typing in the editable text box. All of which only increases the number of test cases.

Can we do better than a straight cross-product of the many options? After all, font size is effectively unlimited in today's word processors, for example (in Microsoft Word font size is a double, not an integer).

Indeed we can do better. Start by developing sets of equivalency classes. For example, testing just a few fonts from each font family is probably sufficient. Similarly, you may decide to group the font effects into those that affect the placement of a character (e.g., subscript) and those that affect the appearance (e.g., strikethrough). That might give us the following matrix:

Font: [Default], Arial, Helvetica, Script, Symbol, SomeDecorativeFont

Font style: None, Bold, Italic, Bold+Italic

Font size: 0, 1, 4, 8, 10, 12, 14, 24, 60, 250

Subscript: Yes, No

Superscript: Yes, No

Strikethrough: Yes, No

Small caps: Yes, No

Set font via: Text box, List box

Set font size via: Text box, List box

This translates to over fifteen thousand test cases. I don't know about you, but this is many more test cases than I want to execute - or will have time to execute, for that matter! Especially considering that for each test we need to not only verify that the preview window in the font dialog is correct but also verify that the font settings are applied correctly in each of the test cases we came up with in the last quiz. What to do?!?

Pairwise combinatorics is our friend here. In short (see my blog post "The Test Cases Go Marching Two By Two, Hurrah, Hurrah" for the full details), pairwise theory holds that most bugs result from a combination of two parameters. Further, multiple combinations can be tested simultaneously without impinging on each other.

Running our equivalency class matrix through a pairwise tool gets us from fifteen thousand test cases down to just sixty - while still achieving the same coverage. Sixty test cases I can deal with!

You can generate pairwise matrices manually, but it's a chore. My blog post on this topic lists several tools that will do the work for you. Not listed there, since it has just recently become publically available, is PICT, the pairwise tool we use internally at Microsoft. Whichever tool you use, pairwise will help you tame your giant test matrices.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...

You have tested across multiple CPU configurations. A common pitfall I see over and over is having every last developer and tester use exactly the same make, model, and configuration of computer. This is especially prevalent in computer labs. Yes, having every machine be exactly the same simplifies setup, troubleshooting, etc. etc. But this is not the world in which your customers live!

This holds true even if you're writing line of business applications for a corporation where the computing environment is tightly controlled and locked down. That "standard" configuration will change on a regular basis, and it will take months or years to complete the switch over. (By which time the standard of course has moved on!)

So be sure to sprinkle the following throughout your development organization (by which term I include dev, test, PM, docs, and anyone else that helps create your app):

• Processors from multiple vendors (i.e., Intel and AMD if you're building Windows applications)


• Multiple versions of each brand of processor (e.g., for Intel, mobile and desktop Celerons and Pentiums)


• Single-, dual-, and multi-processors


• Single-, dual, and multi-core processors


• Hyperthreaded and non-hyperthreaded processors


• Desktop and laptop configurations


• 32-bit and 64-bit configurations

Posted by The Braidy Tester at 07:30 AM  Permalink |

Today's link is Testing Computer Software, by Cem Kaner, Hung Quoc Nguyen, and Jack Falk.

This book taught me testing. If you're just starting to learn about testing, you need to read this book. If you already know a lot about testing, this book is reviewing. If you don't care about testing and found my blog by accident, read this book to understand why bugs escape out to annoy you!

Posted by The Braidy Tester at 07:30 AM  Permalink |

Recently I had a pile of stuff to donate to charity. Their phone center wasn't open when I called to set up a pickup, but their phone message said I could set one up online. Sweet!

Working through the multi-step process I reached a page asking what I was donating. This page had a drop-down list of common items with a corresponding Add button, a text box for free-form entries and its corresponding Add button, and a multi-line text box labeled "Donations". I entered all my entries in the Donations box and then clicked Next.

Only to be confronted with a message that "You cannot modify this field directly". Dismissing which cleared all my entries.

Aargh!

So I tried selecting "Yard sale cleanout" from the common items drop-down. The result: "We are not able to accept the following items in your area: Yard sale cleanout".

So why did you give me the option?

Given that it was the only option left, I entered my items one by one in the free-form text box. This finally worked. (Although I found that entering things it doesn't recognize results in a generic message listing six different items they won't accept in my area, none of which have anything to do with the item I entered. But if you preface that same item with a number it's accepted just fine.)

After selecting a pickup date I entered my phone number and last name. That took me to a page that linked to my donation history! (Which, I later discovered, is also available directly from a link on the site's home page.)

Given all the problems I'd found I was somewhat surprised to see that they actually used a single database for both online and phone-based pickups. But that only makes this security hole that much worse. Given just a name and a phone number - easy to get from the White Pages - I can access the donation history, address, and full contact information for anybody I want.

Oops.

This is a perfect example of why testing is important. I imagine that this site was created by a volunteer developer, and I assume (hope) that that developer did do some testing. But I find that developers want to make sure their code works the way they want it to, while testers want to see how many different ways they can make that code misbehave. And crash. And melt down.

Good developers value their testers because of this. Do you?

Know a good story? Email me and I'll highlight it here!

Posted by The Braidy Tester at 07:30 AM  Permalink |

Tester puzzle time! How would you test the "select a font" dialog in a word processor?

Contents vary across applications, so let's say the dialog contains the following:

• List of available fonts linked to an editable text box


• List of available font styles (e.g., bold, italic) linked to an editable text box


• List of available font sizes linked to an editable text box


• Check boxes for various effects (e.g., strikethrough, small caps, subscript, superscript)


• A preview window that shows the end result of the currently selected options

How many different test cases can you come up with? Put your answers in the comments, or email them to me. I'll summarize and answer next week!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...

You've checked that all text is actually text and not a bitmap or video. Text rendered as a graphic is problematic for two reasons. First, accessibility clients such as screen readers can't see into bitmaps, videos, and animations, so any text embedded in such a graphic is invisible to anyone using accessibility features. Second, graphics with embedded text vastly complicate the localization process. Translating text simply requires modifying the application's resource files, but translating bitmaps and videos requires recompositing them.

If you must place text in bitmaps, you can mitigate your localization pain by creating the bitmaps dynamically at runtime using resourced text strings. Videos and animations may be runtime creatable as well depending on the toolset you use.

As for accessibility, ensure that the relevant information is available some other way: in the supporting text, by closed captioning, ALT tags in HTML, and so on.

Posted by The Braidy Tester at 07:30 AM  Permalink |