Tester Puzzle Answers

Last week's quiz was testing the "select a font" dialog from your favorite word processor.

Tony submits that "since the font is mandatory while the others such as style and size are options [the answer is] (number of fonts) x (number of font styles + 1) x (number of font sizes + 1) x (number of effects + 1). So for example if there are 5 fonts, 3 styles (regular, bold, italic), 5 sizes and 2 effects (strike through, underline) then the number of test cases is 5 x 4 x 6 x 3 = 360."

Tony is close. He omitted combinations of font styles (that is, you can have both bold *and* italic applied) as well as combinations of effects. Missing also is any mention of selecting the font and font size via the list versus by typing in the editable text box. All of which only increases the number of test cases.

Can we do better than a straight cross-product of the many options? After all, font size is effectively unlimited in today's word processors, for example (in Microsoft Word font size is a double, not an integer).

Indeed we can do better. Start by developing sets of equivalency classes. For example, testing just a few fonts from each font family is probably sufficient. Similarly, you may decide to group the font effects into those that affect the placement of a character (e.g., subscript) and those that affect the appearance (e.g., strikethrough). That might give us the following matrix:

Font: [Default], Arial, Helvetica, Script, Symbol, SomeDecorativeFont

Font style: None, Bold, Italic, Bold+Italic

Font size: 0, 1, 4, 8, 10, 12, 14, 24, 60, 250

Subscript: Yes, No

Superscript: Yes, No

Strikethrough: Yes, No

Small caps: Yes, No

Set font via: Text box, List box

Set font size via: Text box, List box

This translates to over fifteen thousand test cases. I don't know about you, but this is many more test cases than I want to execute - or will have time to execute, for that matter! Especially considering that for each test we need to not only verify that the preview window in the font dialog is correct but also verify that the font settings are applied correctly in each of the test cases we came up with in the last quiz. What to do?!?

Pairwise combinatorics is our friend here. In short (see my blog post "The Test Cases Go Marching Two By Two, Hurrah, Hurrah" for the full details), pairwise theory holds that most bugs result from a combination of two parameters. Further, multiple combinations can be tested simultaneously without impinging on each other.

Running our equivalency class matrix through a pairwise tool gets us from fifteen thousand test cases down to just sixty - while still achieving the same coverage. Sixty test cases I can deal with!

You can generate pairwise matrices manually, but it's a chore. My blog post on this topic lists several tools that will do the work for you. Not listed there, since it has just recently become publically available, is PICT, the pairwise tool we use internally at Microsoft. Whichever tool you use, pairwise will help you tame your giant test matrices.

Posted by The Braidy Tester at 07:30 AM  Permalink