A Test of Patience

Every year, we here at Dr. Dobb's put together a DVD archive. It's an amazing resource and, in my admittedly biased opinion, well worth the price. But we don't get there without some gritted teeth. The hardest part about it in recent years has been getting the search functionality right. It has taken us a very long time to settle on a technology that works properly (in case you're interested, it's a custom-tweaked version of JSFind). Breaking in this search technology proved to be my first real exposure to testing.

So why is a search tool so hard to test? Because the potential inputs are infinite. It's no good just making sure the search finds the terms you know you want it to find. You have to make sure that it behaves appropriately for all negative results as well. On one of our first few test search indexes, throwing random nonsense at the search engine yielded tantalizing results: Typing in "blemonge" threw the engine into a fatal tailspin, but "blort" returned the expected "Nothing found" result. Turns out that the search would predictably fail when it traveled down certain branches of the balanced tree of XML files that constituted the search index.

Another issue was browser variance. This is, after all, an entirely HTML-based product, meant to be view with your favorite browser. Well, as it turns out, that means your favorite browser off of our list of supported browsers. Javascript behaves differently enough across the various browsers that some problems were just not fixable. In the end, we managed to get code that ran fairly well on IE and Firefox on Windows, Firefox on Linux, and Firefox and Safari on the Mac. But the browser-specific branches in the Javascript that had to be added boggle the mind.

I'm certain we weren't following good test methodology - in the end, however, we got there with a combination of dogged persistence and creative thinking. I hesitate to say the result is bug-free (what is?), but for the past year, it seems to have worked fairly reliably for our users. So the proof is in the pudding, I guess.

Posted by Kevin Carlson at 12:58 PM  Permalink