THE BOOK OF TESTING

Thoughts From a Braidy Tester

June 2007

You may be familiar with Payson Hall if you read Better Software magazine, or Computerworld, or the Journal of the Institute of Chartered Financial Analysts of India. Or maybe your company was one of the innumerable hardware and software systems integration projects he has consulted on and executed on over the past twenty-five years. Or maybe you have participated in the classes and seminars on systems integration and project management and risk management Payson has been leading at various seminars and conferences for many of those years.

Systems integration, project management, and risk management may not seem to have much to do with testing. If so, take a moment to look at your product through those points of view, and then see if you don't approach testing your product differently. I find Payson's thinkings help me understand these different points of view, which in turn helps me find and prevent issues I likely would have missed otherwise. Which seems all goodness to me!

Here is what Payson has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
PH: My first introduction to testing was to be hired as the part time tester for a commercial product about to be released by a two person company. I was pretty green. I hadn't finished my computer science degree yet, and my notion of testing was "try to find a systematic way to beat on the keys and see if the product breaks". Four years later, as a product manager at the same company (now with 30 people), we had developed a much more rigorous process and the goal of testing was to exercise all aspects of the code (clear box testing) to assure correct behavior and graceful failure if something went wrong. We built stubs and drivers (and sometimes simulated or modified operating system components) to generate every error condition that we could think of. Much of our code was operating system level (real time process control, low level I/O), at that time I was convinced that basis path testing (testing all possible paths through individual modules and integrated subsystems) was the only "true" path to testing. Five years later, I was working on systems that did not lend themselves to basis path testing - too much code was being run from libraries, we were no longer running stand alone systems, and frankly I was a little overwhelmed. The fallback plan at that point was to both assure that we got the intended functionality, and also to try and identify tests that broke the systems. The difference was that the box wasn't clear anymore... sometimes white, sometimes black. This was about the time I met Cem Kaner and James Bach, who were taking testing very seriously. This is also when I made the transition away from hands on development toward project management.

DDJ: What is the most interesting bug you have seen?
PH: The reason I chose to pursue computer science rather than electrical engineering, was that I found the real world messy. I liked the mathematical purity of software, assuming that it would either work or not work as a direct result of my skill and thoroughness as a developer or designer. Consequently, the most interesting (and humbling) bugs to me were probably bugs that I found in operating systems and compilers which I assumed would behave consistently and correctly. Foundation component bugs are particularly nasty, because once you find one the rookies on the team begin assuming that ANY bug they encounter is probably in the operating system or compiler or comm routine and it is difficult as a team leader to help people sustain the appropriate levels of skepticism.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
PH: I would give testers the same guidance that I give analysts and project managers. I think the most important thing is to learn to make a clear distinction between what you KNOW and what you THINK you know. The ability to recognize and capture assumptions before you mistakenly move them into the "truth" category is an essential skill. Recognizing and documenting assumptions is a critical skill because there is rarely time to exhaustively explore any question. Assumptions let us skip areas we believe are safe as we progress on our inquiry, but capturing them as assumptions reminds us that we may need to go back and revisit them systematically if we hit a dead end. You know someone is having an assumption problem when something happens and you hear them say "This is impossible! This CAN'T be happening!". Denying reality suggests that your mental model includes some assumptions that are incorrect.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PH: I don't imagine this will be an original answer, but I think that the exploding complexity of the systems that we must build and test and manage is going to be the greatest challenge we face. I've been building software since 1975. I've been doing it for money since the early 80's. Systems we thought were complex in 1980 consisted of a million machine instructions. Today, systems consist of dozens or hundreds of components of comparable complexity. Some are part of the application, some are part of the runtime environment, some are part of the foundation (OS, comm layer, DB layer, security infrastructure). I don't think that we have dramatically improved software quality much during the last 25 years. I think Cem Kaner is the one who pointed out to me that our skill designing and testing systems is rising linearly, while the size and complexity of the systems we build is rising exponentially. This is a daunting realization that makes me happy that I'm concerned with coaching and training project managers these days rather than taking responsibility for assuring system quality in modern systems. The scariest bit of technical graffiti that I ever saw said: "COMPLEXITY DECAYS". Without a major improvement in the way we conceptualize and conduct system design, implementation, and testing - I must imagine that the whole thing is going to collapse under its own weight sometime soon.

DDJ: Is there anything else you would like to say?
PH: Perhaps in light of the previous answer, I would like to say, "don't give up". I think that the challenges ahead are daunting, but we need to think of new and better ways to build and test systems so that the collapse that appears inevitable can be avoided. From a design perspective, this means partitioning systems into components that are well defined and defend themselves against errors and contamination from adjacent and dependent systems. We must get better about designing systems for testability and ease of diagnosis. A surprising number of organizations today are designing and testing in much the same way that they were 25 years ago. These processes MUST be replaced by more thoughtful approaches. It is easy for organizations to think that what worked last year will work this year and next year. The constant increase in complexity that we witness in every release of everything suggests that simplified thinking (test a few cases to make sure the new version still works) is doomed. I think the leaders in the field get it. The average practitioner in the field is falling further behind each year. We need to reach out to the average and below average performers and help them improve their practices if we must interact with their systems. Don't give up on your efforts to improve practice just because it is a daunting task.

I believe there is a mistaken belief that the art of performance testing and capacity planning is not relevant because of the ever increasing capacity of our hardware and networks. The complexity of our systems make performance testing extremely challenging, but identifying and resolving sources of performance grief is something that is understood by few and always in demand.

Posted by The Braidy Tester at 07:30 AM  Permalink |

I first met Alan Myrvold a few months ago when he emailed me a question about one of my MSDN blog posts. I looked him up in our internal address book to see who he was. "Security Senior SDET" it said. "Hmmm", I thought, "he must know his stuff." Some time later I met him in person at a meeting of our internal Test Architect Group and I learned that, indeed, he does know his stuff.

For the last two years Alan has been a security tester for Microsoft Office. The sixteen years before that he worked at several largish Canadian software companies doing testing, programming, and management. He's a tester again now because he's found he enjoys that most. Here is what Alan has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
AM: My first introduction was a summer job in 1987 helping with quality assurance, control charts and defect prevention at an injection molding plastics factory. Although it wasn’t software, It made me realize that there are many dimensions to product quality, and that even the best testers can miss important issues, like the grill we made that didn’t fit right on the car. I was always addicted to software, and in 1988 joined a software company initially working on quality assurance, including software metrics and policies and procedures for code reviews. Later I was a test automation toolsmith, creating tools for automated testing. I finally discovered the joy of testing while testing statistical software for data mining. I was strongly influenced by the book Testing Computer Software, which turned on a light and changed how I thought about testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AM: Every time I arrived at a firm belief about software testing; that testing is all about verifying customer requirements, or that all tests should be automated, or that no tests should be automated, or that testing is all about finding bugs, or that defining expected results isn’t important, I was later surprised to realize that my belief wasn’t correct in all contexts, and was a dangerous assumption in some.

DDJ: What is the most interesting bug you have seen?
AM: There have been many. I still get very excited when I discover a cool bug. I get excited not only about bugs, but about how to best describe them clearly and accurately. I’ve been doing security testing, and some of most interesting bugs to me are subtle issues that allow cryptography to be bypassed. I found my most recent cool bug while stepping through code in a debugger, and then I just needed to create test data to make the suspect code path get hit. It was a well tested area, and a very subtle bug that allowed validation to be bypassed. I was euphoric when I found it. After logging it, I updated the repro steps and title a bunch of times to make the bug description clearer. It was late in the release, so the bug went through a formal triage. When the bug was being discussed in triage, my team lead pointed out how many times I updated the bug details, and that it must mean I felt strongly about it. The bug was approved for the release, but I had updated the repro steps yet again while they had it open, so they needed to refresh it to make the change.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
AM: Code coverage tends to be overemphasized. Many cool bugs are errors of omission, not commission, or require interesting data or perceptive evaluation of the product’s behavior.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AM: Learn about testing. When interviewing testers, I’m surprised by how few have ever read a book about testing. Testing is not yet well taught in most computer science programs, and testers need to learn how to test. Many developers don’t understand the role of testing, or toss code over the wall with obvious mistakes. Testing doesn’t ensure quality, and doesn’t find all the bugs. I’d like developers to be terrified of bugs escaping to customers. Many small software development companies produce great software without an dedicated testing group, and developers should be just as cautious when there is a test group.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
AM: After moving here from Canada, what foods do you miss most?

DDJ: What would you answer?
AM> Timbits, poutine, and Montreal style bagels.

Posted by The Braidy Tester at 07:30 AM  Permalink |

I first met Elisabeth Hendrickson several years ago at SDWest, where she gave a workshop on testing. When she asked who in the audience was a tester, I was the only person who raised my hand. Then she asked me what I was doing there - she didn't think I would learn anything, I suppose. Which turned out to be patently not true, as I had expected. I especially like her Nouns And Verbs technique for generating test cases. And her Nightmare Headline Game. And...well, rather than enumerating every last one of Elisabeth's ideas, I'll simply point you to her Ruminations blog so that you can find your own favorites.

If Elisabeth's blog whets your appetite for some more personal assistance with testing, you can catch her at SD and other conferences. Or she would be happy to take your money and work with you in the flesh. (Only please don't introduce her to your workmates as "the expensive consultant from California who used to work for a company whose software sucked", as one manager did!) Here is what Elisabeth has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
ESH: My very first introduction to software testing was at a tiny startup in the 1980s. I was a student at the time, and I’d been hired to hunt bugs. That was the company’s test strategy: hire students and pay them a bug bounty for every bug they found. One problem: I didn't know what I was doing. I didn’t understand the technology or the domain, and I knew nothing about testing. I think I found one bug in the whole time I worked there. I don’t think the company knew what they were doing either (ultimately, like many startups, the company went out of business). So when I cite my testing experience, I don't usually count that particular job.

I did learn one thing on that job that stayed with me. I was trying to do something in the system, and I couldn't. "Oh," I thought. "I must just not have sufficient permissions." Later, I discovered that permissions had nothing to do with it. The problem was with a setting in the operating environment.

That experience taught me to look for multiple possible causes for any given observed behavior. I can't assume that my first guess about a problem is right.

Unfortunately, like many lessons I needed to learn, I had to relearn that lesson many times before it stuck.

Some years after that experience, I was one of the more senior testers in yet another startup. In one particular bug report, I included a guess about the underlying problem in the code. A week later, one of the more junior programmers on the project approached me cautiously. "Elisabeth," he said. "You indicated in this bug report that the problem is in the database connection library. I've been all through that code, and I can't see the problem. Could you show me?" The programmer was quite shy about asking. I think he was afraid to look stupid.

I felt terrible, and I was the one who should have been more concerned about looking stupid. I'd made an assertion about the underlying problem without any real evidence. I guessed. It was totally irresponsible on my part, and I'd wasted days of this guy's time. Once again, I'd fallen into the trap of assuming that my first impression about an underlying cause was accurate. The problem turned out to be somewhere completely different.

I hope the lesson has finally stuck, but even now I have to be vigilant to make sure I don't repeat the error.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ESH: The most important thing for any tester to do is connect with their stakeholders: the people who will use the information that testing reveals to make decisions about the project and the software.

I once met a group of testers who had no idea why they tested. They just did. They'd been hired to test, so they tested. They wrote test reports no one ever read and filed bug reports about defects no one ever fixed. They were a very frustrated group of people. They didn't know why no one seemed to be listening to them. When I suggested they talk to their stakeholders, they realized that they didn't even know who in the organization should care about their work.

We talk a lot about requirements for software. Few testers think through the requirements for their testing. And yet there are requirements for testing. There are pieces of information the stakeholders need to have and that testing can provide.

Consider the VP in charge of deciding whether or not to ship the software. She needs to know if the software does what she promised the customer it would do. She needs to know that it doesn't do anything that would harm the customer. And she needs to know that the tests were sufficiently comprehensive that they would have revealed any problems if they existed.

A test group that focuses on reporting bugs won't be able to tell this VP everything she really needs to know. Bug reports tell us what isn't working, but don't tell us anything about what is working.

A test group that focuses on test status reports with pass/fail information won't be able to give this VP the information she needs either. Test pass/fail statistics are usually completely meaningless by themselves. What does it mean to say "We ran 543 tests on the Foo feature with a 87% pass rate?"

To serve this VP well, the test group has to work with the VP to understand what information she'll value, how often she wants to receive it, and what format she'd like to see it in.

And the test group also has to realize that this VP is only one of their stakeholders. The developers also need information from them. Technical support may need information as well. And the technical writers. We testers are in the information business, we have numerous "customers" for the information we provide, and we should do our best to delight our customers with the quality, quantity, frequency, and overall usefulness of the information we provide them.

Knowing about system analysis, test design, test automation, or about the particular domain are all fine and well. But testers who don't know what information their stakeholders need will find that all those other skills don't add up to a whole lot.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
ESH: I think that "repeatability" is overrated. I started hearing the "Testing should be repeatable" mantra in the late 1980s, along with "Testing should be independent" and "If it isn't documented it didn't happen," and other great sayings that work better on a bumper sticker than in reality.

The "repeatability" requirement for testing particularly bothers me because I can think of so few situations in which repeatable is the most important attribute for a given set of tests. I'd rather tests provide useful information about the capabilities and limitations of the software than that they be repeatable.

I think repeatability became so important because it increases manageability. If you execute the same 5246 test cases on every build, it's easier to predict how long testing will take, it's easier to compare the results of two test runs, and it's easier to track progress against a plan. The problem is that none of those things make software better. They may make the process more predictable and measurable, but they don't make the software better. More information can make software better. So I think it's more important that tests be informative than repeatable.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ESH: Learning how to integrate their efforts with the rest of the team. It's not as easy as it sounds.

Traditional testing practices involve designing tests in parallel with the software development, not in collaboration. Testers are accustomed to working on test design, documentation, and execution in isolation. Some have become comfortable with that isolation. They like being able to work independently. They prefer not to have to explain in great detail how they designed a given set of tests.

The problem with this approach is that too often testers end up re-inventing the wheel...or, in our field, re-inventing the structured programming language. We waste time reverse engineering the software to figure out how to test it. We struggle to build test automation exclusively through external interfaces that weren't designed with test automation in mind. We spend time testing conditions the team does not care about and miss cases that the team does care about.

I'm a member of the Agile community and a huge fan of Extreme Programming and Scrum. Agile teams work as a whole, integrating their efforts, and collaborating on the end result. On Agile teams, QA/Test works with the team to move the project forward rather than gate its passage. That's a huge mindshift. Testers on Agile teams have to learn to provide feedback instead of criticism and to apply their testing skills much earlier in the process.

Some have expressed their concern to me that the "integrated team" approach means that testers have to learn to code. I don't think that's true. I think that testers have to bring something to the table that the team values. The developers already know how to code, so someone who specializes in testing probably isn't going to add much in the way of development expertise. Instead, the testers should bring domain expertise and test expertise to the party.

On integrated teams, testers help the other team members learn more about the domain and about testing. They do more than test: they help the team define acceptance criteria, collaborate with developers and business stakeholders to define tests, collaborate with developers to automate the tests, and support the team as a whole by helping the team figure out what "done" looks like and how they'll know when they've achieved it.

Not all testers are comfortable taking such a proactive role, and that's why being a full fledged member of an integrated team is a big challenge.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
ESH: You should ask me "Why do so many people consider testing tedious, and why do you think it's fun?"

People think testing is tedious when they see it as a monotonous process of repeating the same actions over and over. Click this button, enter that value, check for this error. It's repetitive. Boring. Mind numbing.

I have a different perspective. I think that if your tests are that repetitive, that boring, that predictable, you should have automated those tests some time ago. Let computers do what computers are good at: repeating sequences over and over. And let humans do what humans are good at: challenging, analytical, creative work.

As a tester, I see the big picture, and I also have to be able to drill into the details. I examine the system from the outside, and it also helps if I understand the system from the inside. Testing requires me to think about how the software will be used, what kind of vulnerabilities it might have, and what might be interesting things to vary. And that means testing gives me the opportunity to use a wide variety of skills: technical skills, analysis skills, communication skills, and organizational skills. As a tester, software feels like a maze to be explored, a puzzle to be solved. I love the intellectual challenge.

Posted by The Braidy Tester at 07:30 AM  Permalink |

If you follow agile, or testing, or agile testing, you have probably heard of Jonathan Kohl. His blog Collaborative Software Testing is a veritable hotbed of spicy topics such as Post-Agilism. I find Jonathan's other writings useful as well, whether he is talking about pair testing with developers or explaining why unit tests do not automatically mean defect-free software.

Jonathan spent his formative years in various Big Corps and now consults his way around Canada and the United States helping companies (including Big Corps) understand that testing is a challenging intellectual craft. Here is what Jonathan has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JK: I was hired as an intern while I was in university by a small software company that specialized in management systems. They relied heavily on mainframes. The applications that were developed supported dozens of operating systems, and there were several programming languages in use at once to facilitate all of the complexities of the client systems, not to mention new technology being introduced constantly that they needed to adapt to. The culture was an intersection of students and new grads who were chock full of new ideas and embracing new web technology development, and the mainframe experts who had seen many trends come and go in software development. I was struck by the seemingly endless opportunities for learning.

My manager at the time was open to us trying out a lot of ideas, and was supportive of self-education. I spent a lot of time working with experienced engineers who taught me about problem solving, and how they managed their careers in the face of ever-changing technology. In my testing work, I had a broad scope for learning. Not only did I spend time with engineers, but I spent time with customer support, technical writers, and (when I could) with senior managers. Everyone had something valuable to teach me that contributed to my testing work.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JK: Quite early on, I realized that good software testing was what James Bach describes as a "sophisticated thinking art". Testing is often thought of as an unskilled activity, undertaken by people who lack programming or other technical skills. My experience proved otherwise, but as I began teaching software testing skills, I found I was teaching people to recognize and use thinking tools a lot more than I was teaching technical skills. Recently, a student I was working with exclaimed: "You are just teaching me how to think and to develop my investigative skills!"

I've found I'm drawing from a lot of different sources of knowledge and problem solving skills I've picked up over the years. I took a few philosophy and logic courses in university, and that training in thinking paid off dividends when testing software. In fact, I'm still surprised as I realize the source of some of the thinking skills I use consistently as a tester. For example, I recently noticed some of the economic myth busting we did in university economics classes influenced a lot of how I think about testing, particularly critical thinking, dealing with very large models with many influencing factors, designing experiments, dealing with masses of data, and coming to useful conclusions. One useful technique in economic modeling was to try to look at a problem in a unique way. If you could come at a problem from a different angle than the mainstream thinking, you were almost guaranteed to discover something new. (The book "Freakonomics" is full of great examples.) This kind of thinking is valuable in testing.

Now at this stage in my career, I'm pleased when I am training testers and can surprise them when they realize they already have skills they can build on as software testers - no matter what their backgrounds are. If they are thinkers, they can probably learn how to be effective software testers.

DDJ: What is the most interesting bug you have seen?
JK: My most popular bug story is an intermittent bug I tracked down in an embedded device, which is described in my "Knee Testing" blog post. I find intermittent bugs in general to be interesting. Since most of my testing now is with web applications, I have found different sources of intermittent bugs that are common in web applications. Often, they are due to frameworks or libraries that we are using in our system that have been developed by third parties. Ajax applications can be a plentiful source of intermittent errors, particularly with cross-browser support, excessive memory usage on client machines, and security issues.

One recent bug I tracked down was due to an Ajax call to a server that reconstituted the HTML for a particular application web page every few seconds. The object that contained the HTML was populated by data that was read from a database, marked up and sent back to the user's machine without their knowledge or intervention. There was a simple error in the code that caused it. It was supposed to clear the object contents after each call, instead, the new contents were appended to the object, which caused the web page to grown in size at a geometric rate. After a while, memory consumption by the user's web browser would skyrocket, and they would have to reboot their machines. It took a while to track down the problem web page, but once I did, I used a tool called Wireshark to see what was getting sent back to the user's machine. Inspection of a gigantic HTTP response that had the same HTML and CSS printed over and over in the message body coupled with some white box code inspection revealed the problem.

Another interesting class of intermittent errors has to do with the object-relational impedance mismatch between an object system and a relational database. Object Relational Mapping tools are a common culprit, as are database drivers. Both of these tools are often overlooked, particularly when problems in those tools are manifested as intermittent bugs at the user interface. Tracking down errors caused at the GUI level of an application stack can be time consuming and frustrating. If you know what to look for, and use multiple interfaces when testing, you can track down these problems much more quickly. OR Mappers can get out of synch with the database, often due to caching errors, and sometimes fail less than gracefully. Database drivers sometimes have problems interpreting the code generated by the mapping tool, or the messages sent back from the database. Making good use of logging when testing and learning how to read the stack traces these failures leave can help immensely.

/Bugs in third party libraries and frameworks are a lot more common than we realize/. The more development productivity tools proliferate, and the more we rely on them to create complex systems, the more we will see these kinds of problems. Sometimes, particularly in the case of OR Mappers, I wonder if we are really solving the real problem, or just offloading the problem on a tool. I'd like to see more work on the other side of that problem. Is SQL the best we can do on the database side? XQuery is at least one alternative, but we can certainly do more.

DDJ: How would you describe your testing philosophy?
JK: I suppose I have several. On a personal level, I am results-focused, pragmatic, and passionate about learning. The learning part is important, for example, if we model software testing on how humans think and learn and apply that to software systems, we get interesting results. Learning also helps us adapt over time. I also tend to avoid extremes in software development ideas and movements because they limit my field of vision by narrowing focus. As a tester, I want to be in control of my investigative viewfinder, and I prefer combining processes, practices, tools, techniques and theories rather than saying "all tests should be automated", or "all projects should follow (flavour of the month process)". I find the most interesting learning experiences and information about the product in intersections between different development and testing concepts, and discover new models and techniques in the process.

I look at testing techniques on a project the way I look at other kinds of complex systems. One metaphor I use is the "well balanced diet of testing" concept. If we only eat one kind of food, we are out of balance and may not have enough nutrients, vitamins and whatnot to fuel our bodies. On software projects, different kinds of testing reveal different kinds of information, and information helps fuel our projects. Just as only eating from one food group can have unintended consequences, I see adhering to only select practices (such as focusing only on automated unit testing) as akin to drinking only prune juice. Sure, prune juice might be good for you, but if you drink nothing but prune juice, you will have some unintended consequences.

On a research or community level, I look to other investigative disciplines to learn new tools for my testing work. I have a friend who is firefighter, a friend who is a nurse, a friend who is a doctor, a friend who is a pilot... the list goes on. When I study how they do their work, and the skills they develop, I am reminded about how naive our models often are when it comes to software testing and software development. For example, my friends who are in mission critical work do not elevate a certain process or tool to the level we often do in software development. They prefer to have many tools and processes at their disposal. I try to teach others about my own learning, and to share my testing techniques, so part of my philosophy about testing revolves around sharing ideas and collaborating.

On a corporate level, I have found that testing thinking can lead to innovation. I sometimes jokingly refer to this as "the power of negative thinking". We don't tend to innovate when we are comfortable and thinking positively about everything. Sure, we need a certain amount of positive thinking, but we also need some balance. I've noticed that the sources of system failures and problems are often a source for new innovations for a company. Instead of wishful thinking, naive positive attitudes and blind faith that our system is going to work as expected, with a failure being bad news, a failure can be a potential source of innovation. This is a shift in thinking. Too much positive thinking about what we are doing can also negate contingency planning, which can have disastrous results. When companies make that shift in thinking, investigative testing is encouraged, and skilled testers are highly valued. Not only does the information they provide help improve product quality, it can help the company strategically. So part of my philosophy is to help organizations find those points of potential innovation, and celebrate how we discover failures and overcome them for the benefit of the organization.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JK: The same challenges we've had for the past five years - learning and developing software testing skill. This isn't as straightforward as it should be. Recently, we've had processes whose creators that have said "there is no tester role on our projects". The Agile movement has caused an explosion in developer testing tools and techniques, which is great, but practices like Extreme Programming have excluded non-programming testers, which excludes a whole world of testing skills and development. It's hard to develop your skill if you aren't wanted or valued on a team. Recently, there has been pressure against non-programming testers from several fronts. Personally, I'm ambivalent about testers needing to be able to program, mostly because I can program, and I have found that to be incredibly valuable as a tester. However, my ability to program is also a source of blind spots when testing. I discourage thinking that all testers should program due to my experience with fantastic non-programming testers.

I was once a lead on a test team of about 15 people. There were two testers who consistently outperformed about a dozen others. One was a programming tester, the other could not program at all, but came from a technical writing background. That writing skill gave her a whole other perspective on testing software that those of us who could program would miss. I want people like that on my test team, but some of the best testers I know have trouble getting work because some teams don't value their skills, simply because they aren't programmers. The "can you program?" fleecing device for testers is not a good indication of testing skill. In fact, some of the worst testers are failed programmers. While programming skill can be important to testers, there are a lot of other skills we can use when testing. Recently, I've been looking more at User Experience ideas, which are another source of great skills we can use as testers.

The "cram then write the exam" certifications are also a challenge to developing testing skill. The only skill most really seem to certify is that you can study material and write an exam successfully. That does take a certain amount of intelligence, but regurgitating testing theory and being a good tester are two different things.

I encourage testers from all sorts of backgrounds to develop their investigative skills. An easy way to start is to find an investigative discipline you know something about, or are interested in, then learn some of the ideas and techniques, and apply them to software testing. After, share your findings with others. What worked? More importantly, what didn't work? What lessons can we learn from the things that didn't work so well? We are barely scratching the surface on testing knowledge now. I can't wait to see what skilled testers create and discover in the next five years.

Posted by The Braidy Tester at 07:30 AM  Permalink |