Debugging Testing

I am currently reading David J. Agans's book Debugging. It would be worth reading just for his stories, but his insights into debugging are interesting and useful as well. As I read through David's nine essential debugging techniques, I realized that they each apply to testing too:

• Understand the system. If you blindly wander around an application sending random input and poking random UI in random ways you will find some bugs. If you take the time to understand the system, however, and build a model of how (you think) it works, you will find many more bugs - and likely more interesting and critical and useful bugs. The Rule Of Three applies here: a mental model of the application will find different issues than does a state-based model, and a based-on-the-code model will find yet others. How many other types of models can you think of?
• Make it fail. Pretty much the tester's mantra!
• Quit thinking and look. Thinking is good - hence "Understand the system" above. The point here is to pay attention to what your application is actually doing, how it is actually responding to your inputs and actions. It can be difficult to step away from what you "know" is happening into neutral observation, but the effort is well worth it.
• Divide and conquer. The number of places an application *might* have a bug is infinite. Likewise the set of sequences of actions that *might* cause the application to fail. Seemingly insurmountable giant heaps of "things to test" can be disheartening, to say the least. As with the fabled person having elephant for supper, the solution is to take one thing at a time. What is the most important thing to check? Or the easiest? Or the hardest? Pick one feature, or vertical slice, or customer scenario, or whatever, and test it. That's one chunk down, and some information learned. Use that information to help you decide which chunk to tackle next. And again, and again. Before long that intractable mountain of testing will look more like a collection of molehills!
• Change one thing at a time. Attacking an application eighteen different ways at once can be fun, but when you find a bug finding the exact repro steps will likely be difficult. I find it much more effective to do one thing at a time, to begin with at least. Once my application can survive single-vector attacks, then I move on to double-vector attacks, and then multi-vector attacks, and so on.
• Keep an audit trail. You never know when you will run into a bug, nor which of the many different actions you've taken are necessary to make it happen again. Keep a log book where you record the general sequence of steps that you take. Use a screen capture program to record a video or series of screen shots as you go through your testing. Records like these will help the next time you find yourself asking "Now how did I make it do that?"
• Check the plug. Those never-can-reproduce bugs that happen just some of the time, for no apparent reason, might be caused by programming errors such as crossed memory pointers, but they may simply be a sign of a loose or bad memory chip or other hardware-related issue. One time my manager happened by while I was trying to determine the cause of my screen sporadically turning red. It took her about two seconds to discover that my monitor cable was loose. My monitor no longer turned red but my face sure did!
• Get a fresh view. If you just can't figure out how to make a bug repro, or you just cannot think of another test to run, change something. Talk to a colleague about it. Take a different approach. Test something else for awhile. Go for a walk. Familiarity breeds blindness; move into unfamiliarity to see again.
• If you didn't fix it, it ain't fixed. If you logged a bug, and a dev marks it No Repro because it was magically fixed by the time they got around to looking at it, by all means mark that bug as Fixed and then determine exactly what fixed it. If you - or even better, the developer - are unable to pinpoint a specific change that fixed the bug, the bug ain't fixed. It may require different steps to reproduce it now, but it's still there. Find it!

Posted by The Braidy Tester at 07:30 AM  Permalink