Five Questions With Jonathan Kohl

If you follow agile, or testing, or agile testing, you have probably heard of Jonathan Kohl. His blog Collaborative Software Testing is a veritable hotbed of spicy topics such as Post-Agilism. I find Jonathan's other writings useful as well, whether he is talking about pair testing with developers or explaining why unit tests do not automatically mean defect-free software.

Jonathan spent his formative years in various Big Corps and now consults his way around Canada and the United States helping companies (including Big Corps) understand that testing is a challenging intellectual craft. Here is what Jonathan has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JK: I was hired as an intern while I was in university by a small software company that specialized in management systems. They relied heavily on mainframes. The applications that were developed supported dozens of operating systems, and there were several programming languages in use at once to facilitate all of the complexities of the client systems, not to mention new technology being introduced constantly that they needed to adapt to. The culture was an intersection of students and new grads who were chock full of new ideas and embracing new web technology development, and the mainframe experts who had seen many trends come and go in software development. I was struck by the seemingly endless opportunities for learning.

My manager at the time was open to us trying out a lot of ideas, and was supportive of self-education. I spent a lot of time working with experienced engineers who taught me about problem solving, and how they managed their careers in the face of ever-changing technology. In my testing work, I had a broad scope for learning. Not only did I spend time with engineers, but I spent time with customer support, technical writers, and (when I could) with senior managers. Everyone had something valuable to teach me that contributed to my testing work.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JK: Quite early on, I realized that good software testing was what James Bach describes as a "sophisticated thinking art". Testing is often thought of as an unskilled activity, undertaken by people who lack programming or other technical skills. My experience proved otherwise, but as I began teaching software testing skills, I found I was teaching people to recognize and use thinking tools a lot more than I was teaching technical skills. Recently, a student I was working with exclaimed: "You are just teaching me how to think and to develop my investigative skills!"

I've found I'm drawing from a lot of different sources of knowledge and problem solving skills I've picked up over the years. I took a few philosophy and logic courses in university, and that training in thinking paid off dividends when testing software. In fact, I'm still surprised as I realize the source of some of the thinking skills I use consistently as a tester. For example, I recently noticed some of the economic myth busting we did in university economics classes influenced a lot of how I think about testing, particularly critical thinking, dealing with very large models with many influencing factors, designing experiments, dealing with masses of data, and coming to useful conclusions. One useful technique in economic modeling was to try to look at a problem in a unique way. If you could come at a problem from a different angle than the mainstream thinking, you were almost guaranteed to discover something new. (The book "Freakonomics" is full of great examples.) This kind of thinking is valuable in testing.

Now at this stage in my career, I'm pleased when I am training testers and can surprise them when they realize they already have skills they can build on as software testers - no matter what their backgrounds are. If they are thinkers, they can probably learn how to be effective software testers.

DDJ: What is the most interesting bug you have seen?
JK: My most popular bug story is an intermittent bug I tracked down in an embedded device, which is described in my "Knee Testing" blog post. I find intermittent bugs in general to be interesting. Since most of my testing now is with web applications, I have found different sources of intermittent bugs that are common in web applications. Often, they are due to frameworks or libraries that we are using in our system that have been developed by third parties. Ajax applications can be a plentiful source of intermittent errors, particularly with cross-browser support, excessive memory usage on client machines, and security issues.

One recent bug I tracked down was due to an Ajax call to a server that reconstituted the HTML for a particular application web page every few seconds. The object that contained the HTML was populated by data that was read from a database, marked up and sent back to the user's machine without their knowledge or intervention. There was a simple error in the code that caused it. It was supposed to clear the object contents after each call, instead, the new contents were appended to the object, which caused the web page to grown in size at a geometric rate. After a while, memory consumption by the user's web browser would skyrocket, and they would have to reboot their machines. It took a while to track down the problem web page, but once I did, I used a tool called Wireshark to see what was getting sent back to the user's machine. Inspection of a gigantic HTTP response that had the same HTML and CSS printed over and over in the message body coupled with some white box code inspection revealed the problem.

Another interesting class of intermittent errors has to do with the object-relational impedance mismatch between an object system and a relational database. Object Relational Mapping tools are a common culprit, as are database drivers. Both of these tools are often overlooked, particularly when problems in those tools are manifested as intermittent bugs at the user interface. Tracking down errors caused at the GUI level of an application stack can be time consuming and frustrating. If you know what to look for, and use multiple interfaces when testing, you can track down these problems much more quickly. OR Mappers can get out of synch with the database, often due to caching errors, and sometimes fail less than gracefully. Database drivers sometimes have problems interpreting the code generated by the mapping tool, or the messages sent back from the database. Making good use of logging when testing and learning how to read the stack traces these failures leave can help immensely.

/Bugs in third party libraries and frameworks are a lot more common than we realize/. The more development productivity tools proliferate, and the more we rely on them to create complex systems, the more we will see these kinds of problems. Sometimes, particularly in the case of OR Mappers, I wonder if we are really solving the real problem, or just offloading the problem on a tool. I'd like to see more work on the other side of that problem. Is SQL the best we can do on the database side? XQuery is at least one alternative, but we can certainly do more.

DDJ: How would you describe your testing philosophy?
JK: I suppose I have several. On a personal level, I am results-focused, pragmatic, and passionate about learning. The learning part is important, for example, if we model software testing on how humans think and learn and apply that to software systems, we get interesting results. Learning also helps us adapt over time. I also tend to avoid extremes in software development ideas and movements because they limit my field of vision by narrowing focus. As a tester, I want to be in control of my investigative viewfinder, and I prefer combining processes, practices, tools, techniques and theories rather than saying "all tests should be automated", or "all projects should follow (flavour of the month process)". I find the most interesting learning experiences and information about the product in intersections between different development and testing concepts, and discover new models and techniques in the process.

I look at testing techniques on a project the way I look at other kinds of complex systems. One metaphor I use is the "well balanced diet of testing" concept. If we only eat one kind of food, we are out of balance and may not have enough nutrients, vitamins and whatnot to fuel our bodies. On software projects, different kinds of testing reveal different kinds of information, and information helps fuel our projects. Just as only eating from one food group can have unintended consequences, I see adhering to only select practices (such as focusing only on automated unit testing) as akin to drinking only prune juice. Sure, prune juice might be good for you, but if you drink nothing but prune juice, you will have some unintended consequences.

On a research or community level, I look to other investigative disciplines to learn new tools for my testing work. I have a friend who is firefighter, a friend who is a nurse, a friend who is a doctor, a friend who is a pilot... the list goes on. When I study how they do their work, and the skills they develop, I am reminded about how naive our models often are when it comes to software testing and software development. For example, my friends who are in mission critical work do not elevate a certain process or tool to the level we often do in software development. They prefer to have many tools and processes at their disposal. I try to teach others about my own learning, and to share my testing techniques, so part of my philosophy about testing revolves around sharing ideas and collaborating.

On a corporate level, I have found that testing thinking can lead to innovation. I sometimes jokingly refer to this as "the power of negative thinking". We don't tend to innovate when we are comfortable and thinking positively about everything. Sure, we need a certain amount of positive thinking, but we also need some balance. I've noticed that the sources of system failures and problems are often a source for new innovations for a company. Instead of wishful thinking, naive positive attitudes and blind faith that our system is going to work as expected, with a failure being bad news, a failure can be a potential source of innovation. This is a shift in thinking. Too much positive thinking about what we are doing can also negate contingency planning, which can have disastrous results. When companies make that shift in thinking, investigative testing is encouraged, and skilled testers are highly valued. Not only does the information they provide help improve product quality, it can help the company strategically. So part of my philosophy is to help organizations find those points of potential innovation, and celebrate how we discover failures and overcome them for the benefit of the organization.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JK: The same challenges we've had for the past five years - learning and developing software testing skill. This isn't as straightforward as it should be. Recently, we've had processes whose creators that have said "there is no tester role on our projects". The Agile movement has caused an explosion in developer testing tools and techniques, which is great, but practices like Extreme Programming have excluded non-programming testers, which excludes a whole world of testing skills and development. It's hard to develop your skill if you aren't wanted or valued on a team. Recently, there has been pressure against non-programming testers from several fronts. Personally, I'm ambivalent about testers needing to be able to program, mostly because I can program, and I have found that to be incredibly valuable as a tester. However, my ability to program is also a source of blind spots when testing. I discourage thinking that all testers should program due to my experience with fantastic non-programming testers.

I was once a lead on a test team of about 15 people. There were two testers who consistently outperformed about a dozen others. One was a programming tester, the other could not program at all, but came from a technical writing background. That writing skill gave her a whole other perspective on testing software that those of us who could program would miss. I want people like that on my test team, but some of the best testers I know have trouble getting work because some teams don't value their skills, simply because they aren't programmers. The "can you program?" fleecing device for testers is not a good indication of testing skill. In fact, some of the worst testers are failed programmers. While programming skill can be important to testers, there are a lot of other skills we can use when testing. Recently, I've been looking more at User Experience ideas, which are another source of great skills we can use as testers.

The "cram then write the exam" certifications are also a challenge to developing testing skill. The only skill most really seem to certify is that you can study material and write an exam successfully. That does take a certain amount of intelligence, but regurgitating testing theory and being a good tester are two different things.

I encourage testers from all sorts of backgrounds to develop their investigative skills. An easy way to start is to find an investigative discipline you know something about, or are interested in, then learn some of the ideas and techniques, and apply them to software testing. After, share your findings with others. What worked? More importantly, what didn't work? What lessons can we learn from the things that didn't work so well? We are barely scratching the surface on testing knowledge now. I can't wait to see what skilled testers create and discover in the next five years.

Posted by The Braidy Tester at 07:30 AM  Permalink