BREAKPOINTS

Test, Debug, Release, Rinse, Repeat ...

October 2006

Every year, we here at Dr. Dobb's put together a DVD archive. It's an amazing resource and, in my admittedly biased opinion, well worth the price. But we don't get there without some gritted teeth. The hardest part about it in recent years has been getting the search functionality right. It has taken us a very long time to settle on a technology that works properly (in case you're interested, it's a custom-tweaked version of JSFind). Breaking in this search technology proved to be my first real exposure to testing.

So why is a search tool so hard to test? Because the potential inputs are infinite. It's no good just making sure the search finds the terms you know you want it to find. You have to make sure that it behaves appropriately for all negative results as well. On one of our first few test search indexes, throwing random nonsense at the search engine yielded tantalizing results: Typing in "blemonge" threw the engine into a fatal tailspin, but "blort" returned the expected "Nothing found" result. Turns out that the search would predictably fail when it traveled down certain branches of the balanced tree of XML files that constituted the search index.

Another issue was browser variance. This is, after all, an entirely HTML-based product, meant to be view with your favorite browser. Well, as it turns out, that means your favorite browser off of our list of supported browsers. Javascript behaves differently enough across the various browsers that some problems were just not fixable. In the end, we managed to get code that ran fairly well on IE and Firefox on Windows, Firefox on Linux, and Firefox and Safari on the Mac. But the browser-specific branches in the Javascript that had to be added boggle the mind.

I'm certain we weren't following good test methodology - in the end, however, we got there with a combination of dogged persistence and creative thinking. I hesitate to say the result is bug-free (what is?), but for the past year, it seems to have worked fairly reliably for our users. So the proof is in the pudding, I guess.

Posted by Kevin Carlson at 12:58 PM  Permalink |

What do you do when you must test 36 different hardware/OS combinations? First, you commit yourself to unit testing, and seriously consider test-first development. But what testing systems out there can handle this sort of load efficiently? Pablo Santos and Francisco J. Garcia of Codice Software face this challenge all the time, and their response was to extend NUnit to parallel operation to support distributed testing. The payoff? Speed. Read all about it here.

Posted by Kevin Carlson at 02:18 PM  Permalink |

Our own Braidy Tester (Michael Hunter) has posted a couple of great interviews with prominent folks in the testing world, and you owe it to yourself to give them a read. First, Michael interviewed Gerald M. Weinberg, who among other things, was part of the team that programmed the space tracking network for Project Mercury. Last week, Michael talked with James Whittaker, author of "How to Break Software."

Posted by Kevin Carlson at 02:17 PM  Permalink |