When code and documentation disagree

When you find that your code's behavior doesn't match the documentation, which one do you change?

I think that most people starting out in programming would change the documentation. The basis for this viewpoint is that the documentation's purpose is to describe the program's behavior so that the programmer can figure out what to expect. If there is a discrepancy, it means that the documentation is wrong; the easiest way to resolve that discrepancy is to fix the error in the documentation.

With experience comes a more sophisticated viewpoint: The documentation is the system designer's contract with the user, and one cannot change that contract unilaterally. The developer's job is to produce software that conforms to that contract; if it fails to do so, it is the software, not the contract, that must change.

A simple example should serve to reinforce this second viewpoint: If a program adds 2 and 2 and gets 5, the way to fix it is not to change the documentation to say that 2 + 2 = 5.

Still further reflection should reveal still a more sophisticated view: Sometimes documentation will contain mistakes, just as software contains mistakes. So when there is a discrepancy, the wisest course of action is to figure out whether the program or its documentation that is in error, and then to fix the one that is broken.

If only reality were that simple. Once upon a time I was working on a project in which we found a discrepancy between code and documentation: A library routine took two parameters, and the documentation described those parameters in the opposite order from the code. If you used this library in the way that the documentation described, your program would surely fail.

This situation gave us a choice of changing the code or the documentation. The argument that eventually won was the one about the documentation being a contract with the user: Users would probably have copies on their shelves of the documentation we had already published, and those copies would persist long afer the code had been superseded. So we corrected the next release of the library to make it match the documentation.

Of course, this decision was a disaster, because it effectively renedered that library routine useless. The problem was that it became impossible to write code that would work with both versions of the library, which meant that every piece of code that used that library routine had now to be written in two versions: one for the old library and one for the new one. Still worse: There was no reliable way of testing which library was in use, so programmers had to select the right version of the code manually.

This situation came about because of several factors.

First, it was impossible to use the program in the way the documentation said it should work. Thus, anyone who wound up using the program successfully did so by violating the documentation.

Second, "fixing" the code changed its previously useful (albeit undocumented) behavior in an incompatible way. So the programmers who had discovered how to circumvent the documentation error found that their programs stopped working.

Finally, there was no way to unify the old and new behavior. That is, there was no way to write a single program that would work with both versions of the library.

In hindsight, I think that perhaps the best way to resolve this problem would have been to have change the documentation and find a way to warn programmers that the original documentation was incorrect. After all, there was no way to follow the original documentation and obtain a working program.

Posted by Andrew Koenig at 11:45 AM  Permalink