Combinatorial Testing: When Six Variables are Better than Two

Sometimes common sense wins out. At least that's what a team of computer scientists and mathematicians from the National Institute of Standards and Technology, the University of Texas, Arlington, and George Mason University have decided.

According to conventional wisdom, most software failures result from simple -- rather than complex -- events. Based on their research, what Yu Lei, Raghu Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence concluded was that even with software that has hundreds of different variables, most failures were caused by interactions between just two variables, although even more failures could result from interactions of up to six variables.

With this in mind, the research team went beyond conventional pairwise testing (that is, examining the interactions between only two variables at a time), and devised a technique called IPOG for testing different combinations of settings in up to at least six interacting variables at a time.

This led them to develop an open-source tool that catches programming errors by combinatorial testing. The tool which generates tests for exploring interactions among the settings of multiple variables in a given program will be available to beta testing in January 2008. The team is looking for beta testers. If you're interested in beta testing the software, contact NIST’s Raghu Kacker at Raghu.kacker@nist.gov.

Once beta testing is complete, the tool will be made available in executable format, and then in source code form as open source. The team expects that the tool will be particularly useful for increasing the reliability of e-commerce Web sites, which often contain many interacting variables, as well as industrial process controls.

Posted by Jon Erickson at 11:08 AM  Permalink