Five Questions With Eric Jacobson

Eric Jacobson started Life After University as a Teacher Of How To Use Software. That gave him copious opportunities to notice the problems his students had using software and the defects in said software which his students stumbled upon. Eventually he started pointing these problems out to developers and it wasn't long after that until his first official testing gig.

These days Eric works for Turner Broadcasting, where he tests the software that manages which commercials and programming content air when. Hey Eric, schedule me a Star Trek marathon this weekend, will you? <g/>

Here is what Eric has to say:

DDJ: How would you describe your testing philosophy?
EJ: After attending Michael Bolton’s Rapid Software Testing course [RST], I would like to think of myself as a covert RST tester. Covert because not many of my managers dig RST.

My approach is typically as follows but each test session depends on the context.

Select a chunk of AUT [application under test] functionality that devs consider testable, pose questions to business people until I am clear on how they expect the AUT to function, pose questions to devs until I am clear on how they intend the AUT to function. At this point, I usually have some bugs to log before even starting a test. Next, I attack said focused chunk of the AUT. This begins by helping the AUT to do what it feels most comfortable doing. Once it has proven itself to work consistently, I introduce variables (one at a time) that become more complex, until the bugs begin appearing. I let the tests generate from my mind as I witness behavior. I use the popcorn heuristic, and when the bugs begin to stop popping, I end my exploratory session. Finally, I break out my scripted tests, mark them as pass/fail accordingly, and add any valuable ones that were missing. These scripts are fragments, and only contain one step (i.e., do this, expect this). I’m a firm believer in James Bach’s mine field heuristic, and I tend to encourage different paths through the mind field via intentional vagueness.

I spend 20% of my time automating new tests (or fixing broken automated tests). Lately, these tend to focus in areas of performance testing popular paths through the UI and measuring time spans of UI experience and straight service calls. I use a home grown keyword-driven framework wrapped around QuickTest Pro. I’m also fascinated by monkey tests that randomly make decisions on what to do and can run indefinitely, though I only have a few of these and they aren’t good enough to find bugs.

Is that a philosophy?

DDJ: What is the most interesting bug you have seen?
EJ: Unfortunately, I saw this one in production...in an internal HR app I tested. After managers began complaining that performance appraisal comments they had written for their employees began disappearing, I decided to quiz the users and do more testing in said area. I quickly realized the employees could actually delete their manager’s hidden comments by saving new comments of their own. This was a small company and at the time we had no backups to restore. Most managers lost all their free form text for their performance appraisals and didn’t know it until they were about to conduct their face-to-face performance reviews. I was asked to send out an apology email to the entire company for missing that little bug. It was humiliating.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EJ: Though I haven’t read anyone else’s response, I’m sure I’m not alone on this one. Yes, writing manual test cases with details (e.g., UI stuff) is always emphasized and I think it is a major time waster. From my experiences, these test case details are typically ignored upon execution because they are wrong. Which begs the question, should I fix test case details and lose test time or leave them as is, which would leave a trail of incorrect information?

Another typically emphasized metric is the percentage of automated tests. My management loves to set goals like 50% of your test cases should be automated. These metrics are silly because they assume there is value in automating 50% of my tests, which there may or may not be. But I guess if I only write 10 tests, I only have to automate 5 of them. Hmmmm...

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
EJ: You should ask what frustrates me as a tester? I would answer, that there is no stopping point. Developers can sit back and admire a completed app, knowing their code can do what the specs say it should. They only need to provide one solution to a problem. Testers can never sit back and admire a completely tested app because they know if they have more time they can find more bugs. Testers are not providing a solution, we are inventing problems. Yet, as testers, we are constantly answering questions like “Are you done testing?”, “Can we ship?”. These questions are the stuff nightmares are made of.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EJ: As WPF, Silverlight, Flash, and other new UI technologies become more common, there will probably be a period where popular automation tools like QuickTest Pro and SilkTest lag behind in their support. My dev group decided not to use WPF last year, partly because QuickTest Pro did not support it and the folks at Mercury were not aware of any plans to support it.

On the flip side, it appears that Test Driven Development will continue gaining popularity for the next several years. This is good for testers! My dev team currently has over 3500 unit tests, which us testers walk through and use as build acceptance tests. It gives us the piece of mind that a huge chunk of the basic functionality is working. We spend less time chasing the boring environment config problems and basic functionality, and more time looking for the fun, less obvious bugs.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink