Statistical Debugging and the Cooperative Bug Isolation Project

Ben Liblit, an assistant professor in the University of Wisconsin-Madison's department of computer sciences, has put together the Cooperative Bug Isolation Project, a project he refers to as "statistical debugging" because it finds bugs in programs via automated statistical analysis instead of laborious manual inspection.

The way the project works is that Liblit and his team distribute specially modified versions of open source software such as Evolution (similar to Microsoft Outlook), Gnumeric (a spreadsheet), Rhythmbox(similar to iTunes), and Spin, a CPU simulator from Bell Labs.

The software in the Project's download area has been augmented with extra instrumentation code that runs along side the application to monitor its behavior. The specific behavior the Cooperative Bug Isolation Project monitors varies depending on how the application was built. In general, Liblit is looking at data values and decisions within the application, and testing them to see if they show unusual patterns. If monitoring picks up something unusual, and the application also crashes, then they may have isolated the cause of a bug.

Clearly recording everything an application does is too slow and could also be a serious invasion of privacy. Instead, Liblit only looks at a very sparse sample of what is going on. The instrumentation they add is not active all the time. It turns on/off randomly while the application runs, with most of its time spent off. That gives Liblit a partial but fair picture of what the code is doing without slowing you down. You can participate in this project without worrying that our extra monitoring will get in your way. All in all, Liblit has posted 192 versions of eight different open-source applications. These special versions monitor their own behavior while they run, and report back.

The system averages just under 3000 new reports per month, and bug rates vary a great deal across applications, he says, with "crash rates" as high as 8 percent in one application to a low of 0.4 percent in another. All in all, the "kill board" has recorded 546 outright program crashes and 11,369 lower-level errors as of April 2006.

The ACM thinks Liblit is on to something. It named Liblit's doctoral dissertation on cooperative bug isolation (from the University of California-Berkeley) as the best computer science and engineering dissertation in 2005.

The real excitement of the project, Liblit says, is that it could dramatically improve the ability to enhance software post-deployment.

"Software developers deploy their programs and rarely hear directly from users, but the poor guy in tech support gets an earful," he says. "That's the only kind of feedback you get; you lob it over the wall and hope it works.

"It has been cynically observed that software developers use their consumers as beta testers," he says. "I think there's a lot of truth to that observation. The problem is consumers are not very good beta testers. They're not very disciplined, they don't keep good records, they never do the same thing twice. My solution is to make them better beta testers."

Posted by Jon Erickson at 04:52 AM  Permalink