More debugging thoughts

My remarks about assertions and exceptions generated a few comments, so I thought I'd pour some gasoline on the fire.

Many years ago I worked in a university computation center where they had just gotten a PL/I compiler that checked whether programs adhered to the rules of the language during execution.

For example, attempting to access a nonexistent array element in PL/I either does the equivalent of throwing an exception in C++ or has undefined behavior, depending on whether you have told the compiler to check array bounds in the section of code that accesses the nonexistent element. Normally, people who cared about performance would not turn on array range checking. However, if you were using the checking compiler, it would check array ranges for you regardless of whether you had asked it to do so.

We had some kind of utility program that people used fairly often. I have completely forgotten what it did, but it doesn't matter. What matters is that it was written in PL/I, and that people had come to rely on it.

Of course, as soon as the debugging compiler arrived, I tried compiling this program with that compiler. I probably don't need to tell you what happened: The program turned out to have so many run-time errors that I gave up trying to find them all. Nearly all of them were index range errors, but--as it happened--for strings rather than for arrays.

In effect, this was a program that worked only by coincidence, but the coincidence was so reliable that dozens of people had used it for many months without discovering a single problem.

This anecdote is an example of a widespread problem. In effect, the program was not really written in PL/I at all; it was written in an extension of PL/I that included the ways in which the compiler happened to behave when presented with certain kinds of errors. Of course, the author of the program was not aware of taking advantage of such behavior; it was only when the compiler was asked to check for the problem that its existence was even suspected.

Ever since then I have wondered how many programs appear to be working correctly when they're really doing nothing of the kind.

Posted by Andrew Koenig at 10:34 PM  Permalink