Random thoughts about testing

I've spent the past few days in a state of computer-upgrade-induced confusion, most of it trying to track down things that don't quite work for obscure reasons. Naturally, this experience has gotten me thinking about testing.

The first observation about testing is that it's a good idea. I remember Alex Stepanov, the instigator of STL, telling me that in his opinion, all algorithms should be proved correct--and then they should be tested. So coming out in favor of testing is like advocating motherhood and apple pie.

But once you've gotten that far, then what? The XP folks advocate writing tests first, and say that once your program has passed all the tests, you're done. If the tests are inadequate to ensure the behavior you want, you should write more tests.

I wish that view were correct, because it would make lots of people's lives easier. The trouble is that sometimes you want programs to have characteristics that are difficult or even impossible to test. For example, one of your system's requirements might be that it is impossible for a bad guy to break into it. How can you test for that requirement? Or for a compiler, the requirements might be that all errors of a certain kind are detected. Again, how do you test for that?

My next few posts will concern some examples and stories about testing that I think will be particularly entertaining.

Posted by Andrew Koenig at 12:25 PM  Permalink