April 27, 2008
Sleep tight, don't let the test bugs bite
Reminder - please update your reader to the new blog home on Dobb's CodeTalk
A while ago I asked "who tests the test":
" One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?"
I would add that this is especially true considering that the same person that writes the code also writes its unit tests
One answer I had was making sure to keep the triad of tests->code->acceptance tests. so that each two test the third. This is especially good since, usually, the acceptance/integration tests are not written by the developer who wrote the code.
One answer I got was that tests are usually simple enough to be self evident and when tests contain complex constructs you should probably test them as well.
Basically, tests can have two types of problems
- Not testing the right thing
- Not testing the thing right
Code+unit tests +acceptance seem to take care of the first type of problems. The claim that tests are simple should take care of the second type of problems.
However I think that it isn't always true, especially when it comes to using the tests for regression. consider for example, the following two incidents I had where the the problems were actually with the tests:
1. We have a component that interacts with resources in different URIs, so when resources wake up they register themselves with that component. Resources register two URIs -> one for control and one for interactions. We've added a business rule that both URIs should be on the same host (naturally a new test for that was also added but that not the point) suddenly a bunch of the old tests started failing. Sure enough, I took a look at the new code, found a bug, fixed it, but the tests were still in the red. In the end it turned out one of the Uris that is registered during the tests had an extra "L" (locallhost instead of localhost) so the tests failed with the new rule - as they should..
2. After writing a piece of code to "just work" it had to be made multi-threaded. I needed to write a test that would make sure the component can handle multiple concurrent requests. Sure, I thought, I'll just run few or even all of the other tests in parallel and be done with it. The only problem was that the other tests were written to be isolated (i.e. new instance per test) running them in parallel had no effect what-so-ever.
What can we do to help us locate problems and bugs with tests?
- remeber that Test code is code - so as you write more tests you should also refine the tests design and refactor the code accordingly. For instance in the first example above, it was easier to diagnose the problem since the faulty setup was done by a single method used by alll the failing tests (Single Responsibility Principle @ the method level).
- Test-First - One of the reasons test-first is beneficial is because you can actually see the test fail before you implement the code to make it work. This helps you make sure that the test actually tests the behavior you want. When the code is already in place it is harder to make sure if the test works because of the change or not. In the second example running all the tests in parallel everything was still green, which triggered me to write specific tests for checking for multi-threading issues.
What do you do?
Posted by Arnon Rotem-Gal-Oz at 07:13 AM Permalink
|
