THE BOOK OF TESTING

Thoughts From a Braidy Tester

October 2006

If your application installs EXEs, DLLs, LIBs, or any other kind of file - which covers every application I've ever encountered - you have APIs to test. Possibly the number of APIs *should* be zero - as in the case of for-the-app's-use-only DLLs, or one - as in the case of an EXE which does not support command line arguments. But - as every tester knows - what should be the case is not always what is the case.

• Verify that all publicly exposed APIs should in fact be public. Reviewing source code is one way to do this. Alternatively, tools exist for every language to help with this - Lutz Roeder's .Net Reflector is de rigeur for anyone working in Microsoft .Net, for example. For executables, start by invoking the application with "-<command>", ":<command>", "/<command>", "\<command>" and "<command>" command line arguments, replacing "<command>" with "?" or "help" or a filename. If one of the help commands works you know a) that the application does in fact process command line arguments, and b) the format which it expects command line arguments to take.
• Verify that no non-public API can cause harm if accessed "illegally". Just because an API isn't public does not mean it can't be called. Managed code languages often allow anyone who cares to reflect into non-public methods and properties, and vtables can be hacked. For the most part anyone resorting to such tricks has no cause for complaint if they shoot themselves in their foot, but do be sure that confidential information cannot be exposed through such trickery. Simply making your decryption key or license-checking code private is not sufficient to keep it from prying eyes, for example.
• Review all internal and external APIs for your areas of ownership. Should they have the visibility they have? Do they make sense? Do their names make clear their use and intent?
• Verify that every public object, method, property, and routine has been reviewed and tested.
• Verify that all optional arguments work correctly when they are and are not specified.
• Verify that all return values and uncaught exceptions are correct and helpful.
• Verify that all objects, methods, properties, and routines which claim to be thread safe in fact are.
• Verify that each API can be used from all supported languages. For example, ActiveX controls should be usable (at a minimum) from C++, VB, VB.Net, and C#.
• Verify that documentation exists for every public object, method, property, and routine, and that said documentation is correct. Ensure that any code samples in the docs compile cleanly and run correctly.

Posted by The Braidy Tester at 07:30 AM  Permalink |

I am currently reading David J. Agans's book Debugging. It would be worth reading just for his stories, but his insights into debugging are interesting and useful as well. As I read through David's nine essential debugging techniques, I realized that they each apply to testing too:

• Understand the system. If you blindly wander around an application sending random input and poking random UI in random ways you will find some bugs. If you take the time to understand the system, however, and build a model of how (you think) it works, you will find many more bugs - and likely more interesting and critical and useful bugs. The Rule Of Three applies here: a mental model of the application will find different issues than does a state-based model, and a based-on-the-code model will find yet others. How many other types of models can you think of?
• Make it fail. Pretty much the tester's mantra!
• Quit thinking and look. Thinking is good - hence "Understand the system" above. The point here is to pay attention to what your application is actually doing, how it is actually responding to your inputs and actions. It can be difficult to step away from what you "know" is happening into neutral observation, but the effort is well worth it.
• Divide and conquer. The number of places an application *might* have a bug is infinite. Likewise the set of sequences of actions that *might* cause the application to fail. Seemingly insurmountable giant heaps of "things to test" can be disheartening, to say the least. As with the fabled person having elephant for supper, the solution is to take one thing at a time. What is the most important thing to check? Or the easiest? Or the hardest? Pick one feature, or vertical slice, or customer scenario, or whatever, and test it. That's one chunk down, and some information learned. Use that information to help you decide which chunk to tackle next. And again, and again. Before long that intractable mountain of testing will look more like a collection of molehills!
• Change one thing at a time. Attacking an application eighteen different ways at once can be fun, but when you find a bug finding the exact repro steps will likely be difficult. I find it much more effective to do one thing at a time, to begin with at least. Once my application can survive single-vector attacks, then I move on to double-vector attacks, and then multi-vector attacks, and so on.
• Keep an audit trail. You never know when you will run into a bug, nor which of the many different actions you've taken are necessary to make it happen again. Keep a log book where you record the general sequence of steps that you take. Use a screen capture program to record a video or series of screen shots as you go through your testing. Records like these will help the next time you find yourself asking "Now how did I make it do that?"
• Check the plug. Those never-can-reproduce bugs that happen just some of the time, for no apparent reason, might be caused by programming errors such as crossed memory pointers, but they may simply be a sign of a loose or bad memory chip or other hardware-related issue. One time my manager happened by while I was trying to determine the cause of my screen sporadically turning red. It took her about two seconds to discover that my monitor cable was loose. My monitor no longer turned red but my face sure did!
• Get a fresh view. If you just can't figure out how to make a bug repro, or you just cannot think of another test to run, change something. Talk to a colleague about it. Take a different approach. Test something else for awhile. Go for a walk. Familiarity breeds blindness; move into unfamiliarity to see again.
• If you didn't fix it, it ain't fixed. If you logged a bug, and a dev marks it No Repro because it was magically fixed by the time they got around to looking at it, by all means mark that bug as Fixed and then determine exactly what fixed it. If you - or even better, the developer - are unable to pinpoint a specific change that fixed the bug, the bug ain't fixed. It may require different steps to reproduce it now, but it's still there. Find it!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have put your menus and command bars through their paces. There used to be a distinct difference between menus and command bars: menus could have submenus and were always text (perhaps with an optional icon) while command bars were never nested and were only graphics. Nowadays, however, menus and toolbars are more-or-less the same animal and can be mixed-and-matched, so that the only real difference is that command bars are typically always visible whereas menus are transient.

• Verify all commands work from menus and from command bars
• Verify each keyboard shortcut works correctly
• Verify built-in commands work correctly from a custom menu
• Verify built-in commands work correctly from a custom command bar
• Verify custom commands work correctly from a custom menu
• Verify custom commands work correctly from a custom command bar
• Verify custom commands work correctly from a built-in menu
• Verify custom commands work correctly from a built-in command bar
• Verify custom menus and command bars persist correctly
• Verify customizations to built-in menus and command bars persist correctly
• Verify commands hide/disable and show/enable as and only when appropriate
• Verify command captions are correct and consistent with similar terms used elsewhere
• Verify menu and command bar item context menus work correctly (although menus on menus is going a bit too far I think)
• Verify status bar text is correct
• Verify status bar text is not truncated

Posted by The Braidy Tester at 07:30 AM  Permalink |

Those of you who are regular readers of DDJ are likely familiar with Scott W. Ambler, as he writes the magazine's "Agile Edge" column. IBMers likely know him from his current role as Practice Leader Agile Development in IBM's Methods group. Others of you may have come to know him, as did I, through his ever-popular talks at Software Development and other conferences. Scott was talking about refactoring databases while most of us were just learning how to refactor our code (see his latest book Refactoring Databases: Evolutionary Database Design to learn more).

Here's the interview.

Posted by The Braidy Tester at 07:30 AM  Permalink |

International sufficiency testing is important for just about any application, but localization testing kinda only matters if you are localizing your application into other languages. The distinction can be hard to remember, but it's really quite simple: international sufficiency testing verifies that your application does not have any locale-specific assumptions (like expecting the decimal separator to be a decimal point), whereas localization testing verifies your application can be localized into different languages. Although similar the two are completely orthogonal.

The simplest way to get started localization testing is with a pseudo-localized (aka pseudoloc) build. A pseudoloc build takes your native language build and pseudo-localizes it by adding interesting stuff to the beginning and end of each localized string (where "interesting stuff" is determined by the languages to which your product will be translated, but might include e.g. double-byte or right-to-left characters). This process vastly simplifies your localization testing:

• It allows every build to be localized via an automated process, which is vastly faster and cheaper than is the case when a human hand localizes.
• It allows people who may not read a foreign language to test localized builds.
• Strings that should be localized but are not are immediately obvious as they don't have extra characters pre- and post-pended.
• Strings that should not be localized but in fact are do have extra characters pre- and post-pended and thus are also immediately obvious.
• Double-byte bugs are more easily found.
• UI problems such as truncated strings and layout issues become highly noticeable.

If you can, treat pseudoloc as your primary language and do most of your testing on pseudoloc'd builds. This lets you combine loc testing and functionality testing into one. Testing on actual localized builds - functionality testing as well as localization test - is still important but should be trivial. If you do find major localization bugs on a localized build, find a way to move that discovery into your pseudoloc testing next time!

Beyond all that, there are a few specific items to keep in mind as you test (hopefully pseudo-)localized builds:

• Verify each control throughout your user interface (don't forget all those dialog boxes!) is aligned correctly and sized correctly. Common bugs here are auto-sizing controls moving out of alignment with each other, and non-auto-sizing controls truncating their contents.
• Verify all data is ordered/sorted correctly.
• Verify tab order is correct. (No, this shouldn't be affected by the localization process. But weirder things have happened.)
• Verify all strings that should have been localized were. A should-have-been-localized-but-was-not string is likely hard-coded.
• Verify no strings that should not have been localized were.
• Verify all accelerator key sequences were localized.
• Verify each accelerator key sequence is unique.
• Verify all hot key combination were localized.
• Verify each hot key combination is unique.

Posted by The Braidy Tester at 07:30 AM  Permalink |

No, not the was-a-Minnesota Viking is-a-Minnesota Supreme Court judge! Alan Page is a Test Architect on Microsoft's Engineering Excellence team. Among other things, this means Alan teaches testers how to be better testers. Not just newbie testers, either; Alan's class for senior testers is ever in high demand. Beyond teaching, Alan's job has him creating and updating courses, working with Test teams to reach their goals, and writing on his blog and elsewhere.

Alan has been a tester nigh on fifteen years. The last eleven of them have been at Microsoft, where he beat on various flavors of Windows, from Windows 95 to Windows CE. Now that he is with Engineering Excellence he is not directly responsible for testing any specific product but rather is charged with helping all of Microsoft test better. [Insert your favorite Microsoft-bashing joke here...] Here's what Alan has to say:

Posted by The Braidy Tester at 07:30 AM  Permalink |

Diligent Reader David asks that I add a few more terms to my Test Taxonomy Decoder Ring:

• Smoke Test. Also known as a Build Verification Test (BVT). A short, fast, and simple test meant to determine whether a daily build is so broken that it's not worth even installing, let alone testing. The term originally comes from the practice of testing an airtight enclosure by setting off a smoke bomb inside it to see if/where it leaks; also from a basic hardware test where the test fails if the device under test catches fire (thus causing smoke, hence the name).
• Acceptance Test. A test which determines whether a product is of sufficient quality to be accepted for...some use. Most commonly used as a synonym for "customer acceptance test" - i.e., a User Scenario Test.
• Regression Test. A test designed to prove that a specific sequence of actions no longer results in a previously found bug. Note that this does not prove that the bug is fixed!
• Performance Test. A test which measures the time required to complete an action, the product's responsiveness in a specific situation. For example: "How long does it take from the time the user clicks OK to the Open Document until the document is fully loaded and usable?"
• Load Test. A test which measures how many users a web site, service, or server can handle while staying responsive and alive. For example: "How many simultaneous users can our website serve before web page load time exceeds three seconds?" or "Verify query response time is under one second given five thousand simultaneous users".
• Exploratory Testing. Designing and executing a test simultaneously, or in a nearly simultaneous spin loop. Manual tests are often exploratory (even when they are supposed to be tightly scripted); I have yet to see an automated test that was truly exploratory. Compare to Scripted Testing.
• Scripted Testing: Executing a test which was designed some time (typically much) earlier, deviation from which may be frowned upon. Most automated tests are scripted tests; manual tests can be scripted as well. Compare to Exploratory Testing.

Note that the distinction between Exploratory Testing and Scripted Testing can be difficult to discern. See James Bach's article "What Is Exploratory Testing?" for more information on the similarities and differences between the two.

Are there terms you would like decoded? Let me know!

Posted by The Braidy Tester at 07:30 AM  Permalink |