THE BOOK OF TESTING

Thoughts From a Braidy Tester

March 2008

Selena Delesie is a photographer and ballroom dancer who enjoys camping around Canada. She's also the Software QA Manager for Intelligent Mechatronic Systems and President of the Kitchener Waterloo Software Quality Association. She accidentally discovered testing back in high school as she debugged her way through programming class. College brought her more testing experience in the form of helping her classmates determine why their programs weren't doing what they were supposed to. After college she continued both the testing and the mentoring for a variety of employers, including a stint at Research In Motion playing withtesting their wireless devices and services.

Here is what Selena has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SD: High school programming class was my first introduction. We were not taught about testing our programs, though of course they had to work. I remember doing iterations of ‘testing’ on my code to get it running correctly. Though I was proud of my creations when completed, I found debugging them to be much more satisfying. Hunting for and identifying problems in software became an enjoyable activity for me the remainder of my computer classes in high school and university. I was often helping classmates debug their software in addition to my own, even through late nights in the computer lab, and really loving it. I imagined myself a software detective, though I didn’t realize I could turn it into a career.

My real introduction occurred when I fell, quite literally, into a software testing job after university. I had applied to everything under the sun to find a job. Somehow my resume ended up in the wrong pile and someone noticed. That someone read my resume and decided to bring me in for an interview on a whim. I got the job, and I was suddenly a software tester. I was excited to discover that I could get paid to do something I enjoyed doing so much. I soon realized that testing could be an art and a craft. The more I tested, the more intrigued I was to find better ways to do it. I was hooked.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SD: It astonishes me that not a single class I had in school taught me how to test the software I was creating. From what I have observed this hasn’t changed very much over the years. Our society relies so much on software, yet hasn’t promoted many of the activities that will help ensure quality software is released. While companies want to release quality products, there are still a lot of misconceptions about what it takes to do so. Software today is extremely complex, so flexibility and a variety of test techniques is key... the 'old school' approaches to software development and testing are not as effective on their own as they once were.

So there is an ongoing need to educate other people about what testing is and is not, and why it is important. Many people (executives, management, development, and even testers) have no idea what testing is about, how much work it is to test software, why it is a valuable activity, and why it is not just an end-of-lifecycle activity. Teaching others about this craft can be frustrating, and it sometimes feels like a lost cause. I continue to do so just to experience those moments when I see the glimmer in someone’s eye that shows me they 'get it'. It’s great to find a network of people who are doing the same all over the world.

DDJ: How would you describe your testing philosophy?
SD: First, systematic yet varied. I have tried a variety of techniques and approaches in my experiences, and I have discovered that they all work in one situation or another. I prefer to plan and organize work for a project as early as I can, and determine which approaches would be most appropriate for it. I have found it important to be flexible as the plan is executed though, as new information and experiences on the project will revise what an appropriate plan of action is. For me it's about understanding when different approaches are most effective, and not being afraid to utilize them.

Second, learn, learn, and learn some more. When I began my career in testing, I knew very little about it. I actively sought opportunities to learn and expand my knowledge on the job and in formal training. While I now make time to read a variety of books, articles, and blogs, my most valuable learning experiences have come on the job and in conversations with people who make me think. I am constantly learning new things and evolving my methods and approaches in testing, which helps keep me engaged and successful in my work. I try to share my love for learning with those I work with in the hopes that they will catch the learning bug too.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SD: Important things for testers to know and do:

• How to look at something from different perspectives. Using the same perspective (or no perspective) will limit the number of solutions to a problem, as well as the number and type of defects that are discovered. As testers, we need to wear different hats and play different roles. Try pretending to be different stakeholders or users of the software. For example, a tester could pretend to be: a product super-user, an elderly person who is uncomfortable with new technology, the VP of your organization, the IT specialist who will support use of the product, someone who speaks a foreign language, or someone who is deaf. Each stakeholder will care about different aspects of the product. An ability to look at the product from the perspective of different stakeholders will help uncover different problems.


• There is no 'right' way to test. There are many approaches and recommendations for software testing, but not any one is the 'right' way in every situation. It is valuable to understand and know how to use as many of these as possible, so when it is time to solve a problem or select a test approach for a particular piece of software, a tester can dive into their 'toolkit' and use the most suitable one. A mechanic would not diagnose all car problems with a single approach. Similarly, a tester should not attempt to find all software defects with a single approach.
  
  Testing with positive and negative functional tests alone is typically insufficient for most software created in this day and age, yet many organizations still believe the contrary. Consider: risk-based testing, orthogonal array theory, persona testing, flow testing, end-to-end system testing, and creating visual test models. These are a small handful of approaches that can be extremely valuable in different situations and will find different types of software defects that standard functional tests would be unlikely to uncover.


• Learning never ends. Learn as much as possible about anything and everything. If interested in software testing, read about it or take courses. Venture into topics such as critical thinking, problem solving, working with other people, detective work, and even the art of magic tricks. These areas, and so many others, will improve ones' abilities as a tester. I personally aim to learn at least one new thing every day. It keeps me fresh and alert, and helps ensure I don’t get stuck in a rut doing the same things.

• Unit testing is important! I know many developers who feel the crunch to get software developed and delivered, and in that crunch unit testing gets skipped. The result though is that the software test team ends up spending much of their test cycle finding bugs that could have been found during unit test, and less time finding other types of (often more critical) bugs.


• Testers are your friend. The last thing a developer wants is to see bad bugs be released in software they wrote. Testers will help uncover some of these bugs on their own, but can do so much more if developers and testers work together. Teach a fellow tester about the software you developed, talk to them about the software design, and even involve them in code reviews. I have worked on projects where developers and testers rarely interacted, and those where they worked closely with one another. The latter experiences were much more satisfying and led to open communication, testers helping find bugs during the design phase (when they are much cheaper to fix), and a more solid product being released.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Robert Musson has been doing software for twenty-five years now. He spent fifteen of those years at Teradyne working on telecommunications products. He also helped deploy the Team Software Process (TSP) there, which experience he enjoyed so much he later joined the TSP Initiative at the Software Engineering Institute of Carnegie-Mellon University. These days Robert is a development lead on Microsoft's Windows Core Security Test Team, where he manages what he calls the "Department of Statistical Distortions". (Wonder what he means by that? See his chapters in The Practical Guide to Defect Prevention for further details. See my review of said chapters elsewhere on DDJ.)

Here is what Robert has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
RM: My first introduction to formal testing techniques was in a MSSE class at IIT. It was the usual stuff: boundary case analysis, the V model, and so on. But I don’t think it gave me a good appreciation for what testing really means and what it should mean. Testing in software is really all about quality control, that is, finding defects in the software. This is a very dangerous view since it leads to the thought that testing is where quality comes from. The mindset then becomes that testing, and quality, are reactive in nature when in fact testing needs to be proactive. We need methods to predict the quality of the products and to focus effort on assuring the product is ready rather than just pounding away until it seems to be ready.

DDJ: What is the most interesting bug you have seen?
RM: I’ve seen lots of interesting bugs that take down critical systems for extended periods. What makes them interesting to me is the simplicity of the mistake. For example, many years ago someone once put a comment in the code with the "/*" format and forgot the closing "*/". This accidentally commented out an error path. The code worked under all except the most extreme conditions, which of course were impossible to test for. The most insidious bugs are typically simply goofs on the part of the developer. Yet our testing effort tend to focus on ever more complicated techniques thinking that it is the complex race condition or logic error that must be discovered. We often forget about simple things to find simple problems. Code inspections, for example, are often thought of as a development step, but they also should be thought of as a form of test, or rather, quality assurance. In the cited example, the problem was found by someone just looking at the code and not by any complex test that reproduced the problem. Testers should be familiar with the details of development simply so they can discover defects by thinking rather than by brute force. That is really the point of modeling and simulation. Think about the issue and discover problems before they occur.

DDJ: How would you describe your testing philosophy?
RM: Testing is not about finding bugs but about proving readiness. If testers are finding bugs, that indicates a problem with developing. The job of testing is to prove the software should be shipped. Unfortunately, the mindset of finding bugs causes us to ship once our test cases have been run. As a result, software does not so much ship as escape. Most teams have no idea how many undiscovered defects exist in the software nor do they know the potential impact to the customer. As a general rule, teams do not understand the potential costs associated with fixing the undiscovered bugs. This is the software equivalent of product warranty costs. I am really big on tying the test efforts to system costs. A typical software company spends at least as much maintaining the product as they did developing it. Large product teams then begin to collapse under the weight of the rework costs. I call this a software black hole from which no useful work can escape. Only by controlling the risk associated with releasing software will organizations be able to control repair costs.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RM: We tend to focus on advanced testing techniques. Of course new test technologies are important, but I would like to see all testers trained in statistical techniques and measurements. Simple data tracking, like defects per class, provide invaluable information but only when proper analysis is done. We sometimes see people react to data because it seems bad. Only when we do proper analysis can we determine anything meaningful. There are many things about the development process that are counterintuitive and cause us to make wrong decisions. The job of test is to aid the decision making process, and in many cases, only the test team can provide the information. Yet we often ignore the data, or else react only to the last data point and forget long term trend information. I equate proper data management and analysis to good engineering. Testers should use good engineering techniques and we might be able to spread that type of discipline to other aspects of development.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RM: Automation. I say that tongue in check, but humor often has some element of truth in it. There is a current push that everything needs to be automated and more automation is better. Yet when the analysis is done and we look at costs and impacts to the system, automation can be counterproductive. We need the right automation and the right amount of it. Too much is as bad as too little and in some cases worse. This is where simulation and modeling becomes a key to managing costs. We found that we were actually increasing costs by simply automating everything, and what was worse, we actually had lower product throughput without increasing quality. That was one of those counterintuitive things I mentioned earlier. Some areas are better left to good old ad hoc testing. Don’t get me wrong, some areas require automation as the only way to assure quality. But simply writing 2 or 3 times more test code than product code does not improve the quality and will likely end up greatly increasing the costs. This is where proper planning comes in and having your best engineers as testers can really cut product costs and increase quality. There is a view that higher quality means higher costs, and that is just wrong.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |