Here is what Elfriede has to say:

DDJ: How would you describe your testing philosophy?
ED: My testing philosophy is to automate, automate, and automate. Manual testing is labor intensive, increasingly repetitive and thus error prone. It is archaic and cannot keep up with the software development advances. Despite the advances in development practices and tools, the goals of accelerating the rate at which systems can be delivered and reducing their costs cannot be met by simply writing software faster without comparable improvement in the practices and tools for testing the software. Software testing techniques are not keeping pace with software development advances. With the increased use of advanced software development tools, languages, and model driven architectures by software providers, software testing is rapidly becoming the "very longest and most expensive pole in the tent" when it comes to fielding new capabilities into production or to the masses. Automated testing is the answer. Automated testing tools and methodologies (such as the Automated Testing Lifecycle Methodology (ATLM) described in my book Automated Software Testing) have been developed and continue to emerge. Automated software quality tools growth is primarily due to businesses’ desire for higher quality, increased productivity, and faster time to market. Open source available automated testing tools are improving and STAF/STAX is one great example of a powerful open source automation framework.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ED: The most important thing for testers (or anyone) to know is that testing requires skill and expertise. Software testing is a profession and should not be considered only a stepping stone into another department, for example into development. Often the misconception exists that anyone who is able to hold a clipboard can be a tester. I have written various books and articles on the skills required to be a tester (for a list of my books, see http://www.effectivesoftwaretesting.com/Books.aspx); i.e. to be an efficient software tester the person really has to enjoy testing, and not only see it as a temporary involuntary assignment; have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, to understand the technical intricacies of the application-under-test in order to be able to automate the testing efforts, b) understand the business, and c) never stop learning and improving on the testing efforts. Developers need to unit test their code and make automated unit testing part of their build processes, which results in the most successful development efforts.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ED: The biggest challenge will be that testers and the test discipline will continue to play "catch-up" with development advances. With today’s processor cores tearing through billions of instructions per second and software builds of n-tier applications consisting of thousands of source files, computer hardware and software technologies have made many significant advances. Advances in development practices, methodologies, and tools which have been enablers in fielding phenomenal new, innovative, affordable, and timely information processing capabilities will continue to make headlines. In this existing paradigm of rapid development and edge of technology advancements, with a focus on reducing development time and creating re-usable capabilities, efficient and thorough testing of the software will be more critical than ever. Testing techniques and technologies will need to be able to keep up with the phenomenal advancements: Automated Software testing and self-testable components are part of the answer to this challenge.

DDJ: What else should I ask you?
ED: Where do you currently work and what excites you about your current job?

DDJ: What is your answer?
ED: I currently work at Innovative Defense Technologies. The exciting part about this company is that our goal is to improve software testing efforts across DOD and commercial markets, for example we are bringing automated testing solutions to mission critical systems at the Navy and soon to other parts of the DOD. IDT as a company is solely dedicated to improving software testing efforts across the board. We are working with edge of technology testing techniques and have some of the brightest leadership and developers assigned to the software automation tasks and efforts. IDT will change how testing gets done in the long run.

[See my Table Of Contents post for more details about this interview series.]

After work Paul's children help him decompress by ensuring he has neither time nor need to focus on any task for more than a few minutes at a time. Once his kids go to bed Paul studies origami and volunteers at Ontario's Perimeter Institute for Theoretical Physics. And still he finds time to review books and university courses, and to contribute to the Software Engineering Body of Knowledge. (Send kudos or complaints, depending on your view of that corpus, to Paul's email above. <g/>)

Here is what Paul has to say:

In High School, I used an Atari 800XL to connect to Bulletin Board Systems by modem. This was the 1980's and before the Internet was publicly available. I learnt a lot about communication protocols and could even whistle 9600 baud! (That means that I could dial a number and trick a modem into thinking that I was a computer just by whistling into the phone. What a geek. Ha, ha.)

Occasionally I would download some software, a game or an application of some kind without any documentation, and I would spend several days playing with it to see what it could do and what I could figure out about it. It was fun, a game: randomly and methodically hitting keys on the keyboard to see what happened, using tools and editors to read parts of the code, and making notes along the way to keep track of what I learnt. I rarely spent more than a week on any particular program before deleting it and moving onto the next one. This was similar to the Exploratory Testing that I do more formally today.

Later I went to university and majored in Physics. I studied and practiced the Scientific Method and problem solving on a regular basis. I became pretty good at modeling and pattern detection to help me "see" solutions in my mind to assigned problems. Essentially, I became very good at General Systems Thinking, a very useful testing skill, although I didn't know that name at the time.

While I was still in school, I held several contract positions as a programmer, and eventually came to work at Delrina in their Technical Support department in the early 90's. When I started there were fewer than 20 people in the department, but it was the right place at the right time and our department grew by four times within a 6-month period as the company experienced tremendous growth and popularity.

At some point, we started encountering more problems in Support than the QA team could help us solve, so we created our own test group within the Support department to help us get some answers quickly while the QA team focused on testing the next release. My thorough efforts in this Support Test group got the attention of the QA Manager. He came up to me one day and offered me a job in the QA team for the following summer, which I accepted.

On my next school break, I returned to Delrina in the QA department and worked on two cool projects: the modem compatibility project, and testing a new Forms-based information management system. In the four months I worked there I tested hardware (modems), managed a network of computers running through a series of automated regression tests, observed real honest-to-goodness Usability Testing, gained my first exposure to test techniques, and helped write automated test scripts to test the next generation Forms system.

So what did that leave me thinking about the act or concept of testing? I thought it was cool. I thought it was fun. I thought about the diversity in how testing was done in this one company and how each of the different ways helped to make a better quality product and experience for the end users. I reported bugs and they got fixed. I interacted with both Programmers and Tech Support people and enjoyed being the go-between. There was a "science" aspect to Testing as I drew upon my modeling and problem-solving skills from my school experiences. There was also an "art" to it as I drew upon my high school hacking hobby to feel my way through a system and find the information I needed.

I was bitten by the Testing bug and I liked it!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
PC: How complex it is to do a good job in testing. How diverse the range of skills are that you need to be successful. How important a role Psychology plays in software development. How many people vastly underestimate the effort required to do a good job of testing. How shockingly little most developers know about effective testing techniques. How little most educational institutions offer their students in the way of Testing instruction. How much the Testing body of knowledge still has to develop. How much debate there is over terminology and certifications. How hard it is to hire a good senior tester.

Testing has a solid basis in the Scientific Method and so it is open to many of the same theoretical, practical and philosophical debates that you encounter in Science. Things like: the nature of observation and bias, control and awareness of variables within your tests, the role of Statistics and Probability in test design and analysis, the Psychology of information presentation (i.e. for increased understanding), the variability in repetition of experiments, validity of assumptions, and so on.

The people who don't understand this generally do bad testing because their superficial view leads them to believe in simplified 'best practices.' Those of us who have a pretty good idea of what we're doing spend a lot of time trying to undo the damage caused by the processes, beliefs and expectations that sprout from these best practices.

I can't just pick one thing. Many things still surprise me.

DDJ: What is the most interesting bug you have seen?
PC: That's a tough one because I think many of them are interesting in their own way. I still have a very fresh attitude towards testing. Every day is a new opportunity to find an interesting bug. And I can't turn it off either. I see bugs everywhere - in the Grocery store, in newspapers and flyers, children's toys, restaurant menus, you name it. The world is a pretty buggy place.

Some bugs just appear to me by accident, some because I go looking for them, while others require lots of careful and detailed prep work and setup beforehand. For example, I once spent several days Performance Testing a web application to determine the maximum message throughput for a queuing component. I discovered that the application really supported 100 times fewer messages than expected. That was one of those "back to the drawing board" moments. Oops.

One of my favourite accidental bugs happened about a decade ago, when I worked for a company that made commercial math software. The software essentially had two parts: the math engine and the GUI. The company's corporate logo included an interesting multi-coloured 3-D graph representing a calculus function. I remember seeking out the mathematical function that produced the corporate logo and adding it to one of our regression test suites. In one development build, I discovered that the math engine had changed in a way that caused the math function to produce a different graph! I think the bug report I created had a title like "We can no longer produce our Corporate Logo with our own software!" They fixed it right away. We all had a good laugh over that one for many weeks afterwards.

One of the strangest bugs I remember happened when I was working in the Tech Support department at Delrina back in the early 90's. We were supporting Fax modem software in early versions of DOS and MS Windows. There was one customer who had bought our software and a supported fax modem but he just couldn't get the two to work together. My colleague who took the call tried everything. He checked and changed the computer's BIOS (this was before Plug-and-Play), added the necessary changes to the Windows 3.x environment, and tried a variety of modem initialization and configuration strings. Nothing worked. He felt bad but had to tell the customer that we just couldn't figure out why the software and hardware didn't work together. Some time later that same customer sent us a fax using our software with the message: "My computer got hit by a power surge during a lightning storm and the modem started working with your software! Thanks for your help!"

That fax was posted in our lunchroom for many, many months. That was funny. Weird too. Not a recommended fix though: modem doesn't work? Zap it with lightning! =D

DDJ: How would you describe your testing philosophy?
PC: I think a testing philosophy has several different aspects to it.

To begin with, I think that Testing is best understood in the context of the bigger picture of what is generally going on in Software Development projects. That is, software development is essentially the process of translating an idea from someone's head into computer instruction sets that perform some tasks useful for some purpose. Time and cost also factor in here, but people are the biggest part of the equation. So, if software development is a creative, people-based translation process, what is testing?

I believe that Testing is a way by which we empirically check the progress of the brain-to-code translation. If done well, Testing provides the feedback required to make the necessary adjustments along the way to ensure that we hit the target at the end of the project. If done poorly or not at all, the information may be insufficient or come too late to be useful in this process. Missing the target may have huge consequences. For example, people may lose their jobs, people may lose their finances, or people may die, to name a few.

How is testing done? It is simply done by shifting your perspective on the creative process, asking questions, making observations, and reporting findings to the decision makers responsible for the product release. I suppose "simply" may not be the best term here since each one of those steps require skill and knowledge in order for you to be effective and successful.

I believe that the Scientific Method and Statistics are cornerstones in effective testing approaches. I believe that you have to learn how to recognize a bug and it begins with how you think about the system and what you think the correct expected behaviour should be. Psychology also plays an important role in Testing. Testers are influenced everyday by many people and distracted by many things, and we need to be aware of the influences and biases that we bring to the job.

Because of the impossibility of complete testing, decision-making and good judgment skills are also important in Testing. Key project factors such as quality, time and cost need to be integral parts of the daily decision-making processes, whether you are trying to decide which features to test (or skip) or helping to decide when to ship the product. "Risk-Based Testing" is a popular approach that people may use to evaluate the importance of their testing effort based on the impact of an event and the probability of occurrence (back to Statistics here). To be useful, these decisions need to be made regularly and not just once per project.

Overall, I see testing as a creative, methodical, investigative, disciplined, fun, people-based way of helping to turn ideas into products. Testing is a service to the development organization, to the creative process of figuring out what should be in the product and how it should work. Good testers generate good, timely information for the stakeholders to use - in time for them to use it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PC: I've watched Agile Development methodologies slowly gain momentum over the last five years. I knew many development managers who were intrigued by the Agile Principles five years ago but were afraid to try the new ways of developing software. I have seen a shift happen in Development as more and more companies are adopting agile practices. That shift has yet to begin with Software Testing.

There's an approach called Exploratory Testing (ET) that has been around for a while but many people have been afraid to try it. Unfortunately, there appears to be much confusion over what ET really means and how to do it. As a result, there are many test teams still doing things the old way, relying upon their trusty test documentation and not realizing that they are making themselves obsolete.

This 'old way' that I am referring to is a documentation-based testing approach where someone creates documents with many, many test cases and then someone (else?) executes those test cases to generate test reports and other documentation. I suppose it keeps the Paper Industry prosperous.

One of the major problems I have with this approach is that it fundamentally follows a "Waterfall" model, and therefore is not compatible with Agile practices and principles. I have already seen some companies struggle to understand why their testers aren't really helping on Agile development projects. Therefore, the quick conclusion must be that testers aren't required on Agile projects, right? Not so!

Exploratory Testing, when performed by skilled testers, may be a very effective complement for Agile projects. It is adaptable, responsive to change, effective and produces exactly the kind of information that developers and stakeholders need on the project.

So, over the next five years, I see several big challenges facing us.

For testers:

1. Testers who wish to remain useful will need to learn new techniques and new skills that will enable them to be more effective, adaptable and responsive to change in a fast-moving industry.


2. Exploratory Testing is going to gain popularity in understanding and usage. This is going to change how test teams work together. It will also change how outside people work and interact with test teams. People who are afraid of change should just get out now. ;)

For the test discipline:

1. More courses in Software Testing and good testing practices in context need to be created and integrated into Computer Science and Software Engineering programs at educational institutions everywhere.


2. New Test Management and communication frameworks need to be developed to help test teams work effectively in a diversity of project sizes. There are very few options available today to help you manage or communicate your ET efforts. More research and collaboration will be required to help the test discipline move forward in a good direction.


3. Most of the vendor Test Management tools are based upon the central idea of a "test case." When that concept becomes obsolete, vendors will have to redesign their tools to be useful to testers again.

In summary, the biggest challenges facing testers are in self-development and embracing new approaches that improve their effectiveness and usefulness. The biggest challenges facing the test discipline are in educating more people in useful testing knowledge and techniques, and in developing new tools and techniques to help us manage the evolving testing practices. Shouldn't be too hard, right? =)

[See my Table Of Contents post for more details about this interview series.]

Can you tell Johanna runs a management consulting business? <g/> Also why she co-hosts AYE and helps Jerry Weinberg teach PSL?

Johanna follows the Helpful Rule in everything she does, and her books, workshops, and consulting practice all help you understand how to best do the Good Job you want to do. Here is what Johanna has to say:

Since I'd been a developer for the previous 9 years, I applied what I knew about development to testing. I did these things, just as I had when I had been a developer:

• Explored the application for a short while to decide where to automate.


• Used combinatorial testing techniques.


• Learned quickly to ask about the vital few scenarios that our customers would use, so I could test those in detail, looking for the places our developers had blind spots.

I'm still not an expert in Lisp, but at one point, I was good enough to write lots of little pieces of code that we could use for tests.

What impressed the developers was how "nasty" I was. I found problems no one imagined could exist. I didn't think I was all that nasty; I was using the product the way I thought a user might :-) The most valuable thing the tester brings to his or her work is the curiosity and skepticism about how the product should/does work. Developers could never believe all I did was try to approach the application with an open mind. "Why did you do that?" "It seemed reasonable to me." "It's *not* reasonable!" And then, if the developers didn't fix the problem, sure enough, a customer *would* find exactly that problem. A number of developers were surprised by how creative testers needed to be.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JR: I am constantly amazed at how easy it is to break systems that provide an interface to the general public. For example, my local grocery store upgraded their software last Friday. On Sunday, when I did my grocery shopping, I whipped out my credit card to pay, and the cashier said, "Oh, if it's a <large-bank-name>, we can't take that today." "Why not?" "The debit part of it doesn't work. You need to use a credit card."

How is this possible? How could the vendor not test (or not realize there's a problem with) the software for a common banking partner? I don't know about the cost to my local grocery store, but you can bet I'll be checking the prices of what I buy. If the vendor can't get the payment details right, how will they get the cost details right? (You and I both suspect that the price lookup is different from the payment verification, but if their vendor got one thing that wrong, why wouldn't the vendor make other mistakes?)

I don't understand how producers can continually *not* test their most common scenarios. These common scenarios aren't just the happy paths; they're the common ways people interact with the system, whether that scenario is a common mistake or a typical job the user wants to accomplish. I'm surprised when testers (and developers) don't break those scenarios down from system level tests to smaller and smaller parts of tests. Knowing that your product works in its most common form gives me a feeling of confidence about the system. And, if I can't prove that the common scenarios work, I have areas I can address in more detail.

DDJ: How would you describe your testing philosophy?
JR: I want to make sure the testers are testing the areas that provide the most value to the customer and the areas where we have the most risk. That means the testers have to work with the requirements people to understand the value to the customer. And, the testers have to work with the developers to understand the areas of risk.

JR: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JR: There are two important things for testers to know: Testers need to know about the systems they are testing and how to be friends with their developers. They need to learn the common product scenarios; the implementation language(s) and how developers can be tripped up by the language; what's most valuable to the customers; how to demand automation hooks from developers; and how to use those hooks to develop small pieces of automation. Testers who ask these kinds of questions also tend to be people who use their previous knowledge of other systems and apply that knowledge to this system.

Testers need to be friends with developers, so they can eat lunch with their colleagues and ask, "What was challenging for you? What should I look at? What should I avoid?" I don't necessarily believe the "what to avoid"--I'll ask why so I can think of ways to test.

Developers need to learn how to test as they go, so they can learn how their code could break. Developers have to learn that they can't just do unit tests; they need to look at larger chunks. There's a huge gap between unit tests and system tests, and most of that gap is the developer's responsibility.

Testers, along with developers, need to continually re-examine their approaches and ask themselves, "Am I learning something new from this testing approach? If not, what do I need to?"

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JR: This whole business of measuring how many defects there are by severity. Oh please. Severity is irrelevant. If you have defects in the code, it's dirty. Seeing one defect means it's ok to have more defects. If testers showed the total remaining open number of defects by week, instead of anything else by week, the trend would have a lot more benefit to the project. Developers need to learn to clean up the code, and the best way to do that is to measure the total number of remaining open defects.

Severity and priority might be useful for organizing when and what to fix, but people start playing the defect promotion/demotion game as soon as they start counting by severity. That game allows people to put blinders on, and release broken software.

Now, it's always a business decision about when and what to release, but the people who make that decision need the data to make a good decision. They can't make good decisions without trend data.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JR: What should project managers know about testing?

Project managers need to know which testing should be done by which people, and if those people are actually performing those tests. A PM who waits for the testers to test the product (of course, in three weeks!) is an incompetent PM. Software is a design problem, not a manufacturing problem. You gotta test the design as you go, which means no matter what lifecycle you use, you have to start testing from the beginning.

Test driven development (TDD) is really about design, and good PMs should encourage their developers to use TDD. But even if the organization is wedded to waterfall, or if they think they need an architectural phase, the testers should know enough to participate in architectural reviews, and design reviews. Testers who bring curiosity and skepticism will help the developers create a much better product.

Sometimes, asking the "dumb" questions help people realize what's going on in the product. I once asked "How does that queue work? I understand how things get into it, but I don't understand how things get out." Turns out, there was a design problem in removing things from the queue. If I hadn't learned enough about systems to know that, or if I had been uninterested in the code, I would never have asked that question.

[See my Table Of Contents post for more details about this interview series.]

Here is what Adam has to say:

DDJ: What is the most interesting bug you have seen?
AKW: Do I have to pick just one? :) A scheduling dialog exhibited some interesting behavior found by Aqiqul Hoda, a tester on my team. The dialog contained a date/time control. If you had the date/time dialog open when the computer time changed to be on the hour an unhandled exception would appear. Aqiqul logged a bug for the unhandled exception without knowing the root cause. All of us thought he was crazy - until he reproduced it later in the afternoon. Eventually we did enough diagnosing and troubleshooting to get to the root cause. It was an amazing bug and great team troubleshooting.

DDJ: How would you describe your testing philosophy?
AKW: My testing philosophy used to be about stopping the shipping of bad software (bad in my opinion). Now my philosophy is focused on managing the confidence and risk of project stakeholders. My primary aim is to provide stakeholders with accurate and timely information so the project team can make informed business decisions about the software.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AKW: One of the most important things for me has been keeping my skills sharp. I'd like people to remember to practice the testing craft not only at work but in everyday life. Let testing permeate your life. Always try to look for new and interesting ways to do things.

I do this by reading a lot and testing random programs that I know nothing about, just for the sake of learning and keeping my skills sharp. I have gotten involved in testing communities and attended dinner meetings and conferences. I also do things that put me outside my comfort zone like improv classes. All of these things link back to my work.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AKW: The biggest challenge I see for testers is finding ways to test smarter. Systems are getting complex with technology, scenarios and interactions. This creates a huge testing space. It's no longer sufficient (and in some contexts never was acceptable) to bang away at the keyboard and hope "bad things happen".

The challenge for the testing discipline is getting fresh, young blood to pursue testing as a career. The testing world tends to get ignored in university curriculums and isn't even considered as a career option by most university graduates.

[See my Table Of Contents post for more details about this interview series.]

Here is what Joshua has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JW: After all the technical work, the code, the bug reports, etc., I would have to say that effective communication can make or break your career. I have made plenty of mistakes, and most of them were trying to get the right message across. Communication with your counterpart developers, the program managers, your lead or your employees can create trust, or it can erode it. Sometimes the unintended message is not just a result of what you say, but of how you say it. There are a wide range of strategies people take up when they communication with others. Some people get aggressive, some grow more passive. Your goal is to deliver the message you have in a way that you accomplish your design, not just to say the words. This matters in all kinds of scenarios from defect reports all the way to defending a bug before the release management team. It especially matters when leading a team. Communication can be tough to master, because it isn't a hard skill, but it often carries more weight than your technical know-how.

DDJ: What is the most interesting bug you have seen?
JW: There was this bug that only occurred because of a mix of power management and removable storage. It was interesting to me because it demonstrated how even good test teams can wear blinders. I am confident that the removable storage team did a full battery of tests, probably achieving incredible code coverage results with their testing of removable storage. The same is true of the power management team. But, they never got together. It isn't anyone's fault, but then again, it's both their faults. Should storage drivers and functionality include power management testing? Should the power management test team have a huge array of different kinds of devices and removable media? Where does the responsibility of one team end and the other begin? There aren't perfect answers to these questions. This kind of challenge is part of what makes testing as a career fun: it's not all black and white. To be effective, we have to look outside our boxes, and see the world a little more like the customer does. Only by doing so can we understand how to best test.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JW: Some of the best testing I have ever been a part of was a single SDET and a single SDE focusing on a specific feature in a tightly-coupled feedback loop. They made the most progress because they both shared the same goals: make the feature and the tests great. They both contributed to both goals, even though they owned only one each. This was great because they could communicate openly and honestly. They put away their egos and worked together. The "Quality" wasn't left to the SDET; they both owned it. Not every project will work out like that; but the core principle doesn't change: put away your ego and remember that everyone is responsible for the quality of the product.

DDJ: Is there anything else you would like to say?
JW: People need to read. Testers need to study other disciplines and understand what other companies are doing. I am lucky to be in a group of avid readers, and it has really pushed me to read more. I have really enjoyed reading books on business theory and process management from industries outside software. It has been remarkably educational and eye-opening in how I view quality. I would challenge testers to read. And, they can start with a book that I contributed to: A Practical Guide to Defect Prevention, with more info at www.defectprevention.org. It has a lot of info about software development, but also has a lot of info about things we have learned from other industries.

[See my Table Of Contents post for more details about this interview series.]

Here is what Selena has to say:

• How to look at something from different perspectives. Using the same perspective (or no perspective) will limit the number of solutions to a problem, as well as the number and type of defects that are discovered. As testers, we need to wear different hats and play different roles. Try pretending to be different stakeholders or users of the software. For example, a tester could pretend to be: a product super-user, an elderly person who is uncomfortable with new technology, the VP of your organization, the IT specialist who will support use of the product, someone who speaks a foreign language, or someone who is deaf. Each stakeholder will care about different aspects of the product. An ability to look at the product from the perspective of different stakeholders will help uncover different problems.


• There is no 'right' way to test. There are many approaches and recommendations for software testing, but not any one is the 'right' way in every situation. It is valuable to understand and know how to use as many of these as possible, so when it is time to solve a problem or select a test approach for a particular piece of software, a tester can dive into their 'toolkit' and use the most suitable one. A mechanic would not diagnose all car problems with a single approach. Similarly, a tester should not attempt to find all software defects with a single approach.
  
  Testing with positive and negative functional tests alone is typically insufficient for most software created in this day and age, yet many organizations still believe the contrary. Consider: risk-based testing, orthogonal array theory, persona testing, flow testing, end-to-end system testing, and creating visual test models. These are a small handful of approaches that can be extremely valuable in different situations and will find different types of software defects that standard functional tests would be unlikely to uncover.


• Learning never ends. Learn as much as possible about anything and everything. If interested in software testing, read about it or take courses. Venture into topics such as critical thinking, problem solving, working with other people, detective work, and even the art of magic tricks. These areas, and so many others, will improve ones' abilities as a tester. I personally aim to learn at least one new thing every day. It keeps me fresh and alert, and helps ensure I don’t get stuck in a rut doing the same things.

• Unit testing is important! I know many developers who feel the crunch to get software developed and delivered, and in that crunch unit testing gets skipped. The result though is that the software test team ends up spending much of their test cycle finding bugs that could have been found during unit test, and less time finding other types of (often more critical) bugs.


• Testers are your friend. The last thing a developer wants is to see bad bugs be released in software they wrote. Testers will help uncover some of these bugs on their own, but can do so much more if developers and testers work together. Teach a fellow tester about the software you developed, talk to them about the software design, and even involve them in code reviews. I have worked on projects where developers and testers rarely interacted, and those where they worked closely with one another. The latter experiences were much more satisfying and led to open communication, testers helping find bugs during the design phase (when they are much cheaper to fix), and a more solid product being released.

[See my Table Of Contents post for more details about this interview series.]

Here is what Robert has to say:

DDJ: What is the most interesting bug you have seen?
RM: I’ve seen lots of interesting bugs that take down critical systems for extended periods. What makes them interesting to me is the simplicity of the mistake. For example, many years ago someone once put a comment in the code with the "/*" format and forgot the closing "*/". This accidentally commented out an error path. The code worked under all except the most extreme conditions, which of course were impossible to test for. The most insidious bugs are typically simply goofs on the part of the developer. Yet our testing effort tend to focus on ever more complicated techniques thinking that it is the complex race condition or logic error that must be discovered. We often forget about simple things to find simple problems. Code inspections, for example, are often thought of as a development step, but they also should be thought of as a form of test, or rather, quality assurance. In the cited example, the problem was found by someone just looking at the code and not by any complex test that reproduced the problem. Testers should be familiar with the details of development simply so they can discover defects by thinking rather than by brute force. That is really the point of modeling and simulation. Think about the issue and discover problems before they occur.

DDJ: How would you describe your testing philosophy?
RM: Testing is not about finding bugs but about proving readiness. If testers are finding bugs, that indicates a problem with developing. The job of testing is to prove the software should be shipped. Unfortunately, the mindset of finding bugs causes us to ship once our test cases have been run. As a result, software does not so much ship as escape. Most teams have no idea how many undiscovered defects exist in the software nor do they know the potential impact to the customer. As a general rule, teams do not understand the potential costs associated with fixing the undiscovered bugs. This is the software equivalent of product warranty costs. I am really big on tying the test efforts to system costs. A typical software company spends at least as much maintaining the product as they did developing it. Large product teams then begin to collapse under the weight of the rework costs. I call this a software black hole from which no useful work can escape. Only by controlling the risk associated with releasing software will organizations be able to control repair costs.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RM: We tend to focus on advanced testing techniques. Of course new test technologies are important, but I would like to see all testers trained in statistical techniques and measurements. Simple data tracking, like defects per class, provide invaluable information but only when proper analysis is done. We sometimes see people react to data because it seems bad. Only when we do proper analysis can we determine anything meaningful. There are many things about the development process that are counterintuitive and cause us to make wrong decisions. The job of test is to aid the decision making process, and in many cases, only the test team can provide the information. Yet we often ignore the data, or else react only to the last data point and forget long term trend information. I equate proper data management and analysis to good engineering. Testers should use good engineering techniques and we might be able to spread that type of discipline to other aspects of development.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RM: Automation. I say that tongue in check, but humor often has some element of truth in it. There is a current push that everything needs to be automated and more automation is better. Yet when the analysis is done and we look at costs and impacts to the system, automation can be counterproductive. We need the right automation and the right amount of it. Too much is as bad as too little and in some cases worse. This is where simulation and modeling becomes a key to managing costs. We found that we were actually increasing costs by simply automating everything, and what was worse, we actually had lower product throughput without increasing quality. That was one of those counterintuitive things I mentioned earlier. Some areas are better left to good old ad hoc testing. Don’t get me wrong, some areas require automation as the only way to assure quality. But simply writing 2 or 3 times more test code than product code does not improve the quality and will likely end up greatly increasing the costs. This is where proper planning comes in and having your best engineers as testers can really cut product costs and increase quality. There is a view that higher quality means higher costs, and that is just wrong.

[See my Table Of Contents post for more details about this interview series.]

Here is what Wayne has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
WCM: I am always surprised how consistently test effort is underestimated in this industry. Fred Brooks said 30 years ago, “…few allowed one-half of the project schedule for testing, but that most did indeed spend half of the actual schedule for that purpose.” I think that’s still very common today. I struggle with understanding the forces that perpetuate it. I believe that just about everyone understands the value of testing early. I believe that just about everyone expects troublesome bugs to appear as the product progresses. Across the industry, software project managers and their teams have years of experience. Why do we – the industry - keep repeatedly underestimating testing effort?

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
WCM: It’s hard to remember my first introduction. I often feel like I’m constantly being introduced to testing. It amazes me how many niche testing fields there are in software development – it’s not just enough to have confidence that the application works anymore (maybe it never was). Once you’ve really gotten involved with it full-time, you see how much there is that you have to look at to certify an app – security, load, endurance, usability – it’s a long list. Maybe that’s why we underestimate it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
WCM: Getting developers to be more rigorous about automated unit testing.

I am a big supporter of the concept of Test-Driven Development (TDD). TDD transforms the health of software and opens the gateway to other kinds of tests and QA. I only have anecdotes, but the projects at ExxonMobil that are more disciplined about their unit tests develop faster and with lower bug rates than the projects that do not.

Talking to vendors, I think it is pretty clear that manual testing is still king in the industry, and that has to change. Our systems are becoming more and more interconnected; we have to be able to reliably test these systems as quickly and efficiently as possible.

I think the prize in Service Oriented Architecture demands disciplined test automation to succeed. I don’t believe that manual testing is going to keep pace managing large interconnected systems that are being modified by many people across many locations and groups. It’s not enough that some of our software teams do this well; I think this is a practice that we need to spread consistently throughout our organization.

DDJ: Is there anything else you would like to say?
WCM: One of the new things our testing groups are doing is to start to take a look at the ergonomics of our applications and workflows.

Today, new hires in their early 20s come to work with 15 years or more of wear and tear on their wrists, neck, and back. We want to ensure our employees can maintain the intensity and enjoyment they have for computing when they are 51 or 61 that they had when they were 21.

I’ve appreciated that ExxonMobil takes this commitment seriously and extends beyond just our own internally developed applications. Ergonomics and usability of our applications is a fundamental priority of senior management. We’re working with vendors to develop more usable, ergonomic applications. We continue to extend our network of internal and external ergonomic experts to devise ways to make literally healthier applications and environments for our users.

For us, there is a lot to do and to learn, but developing more ergonomic applications is a win for everyone. Developers take a lot of pride in the effort they put into their applications. They want feedback that gives their applications graceful UIs. Users love transparent applications that allow them to focus on their business problem and forget the tool. Everybody wins from intuitive, integrated applications where no one gets hurt.

[See my Table Of Contents post for more details about this interview series.]

Here is what Ross has to say:

DDJ: What is the most interesting bug you have seen?
RS: I don’t know if it was the most interesting, but the most memorable one for me was in my first testing role (COM/OLE for the first version of Windows NT). I ran into a case where a mouse move message was not being redirected correctly if it followed a right button click message. It was tremendously obscure and seemed to happen intermittently, but after a lot of time creating convoluted hardware configurations with the mouse plugged, unplugged, and watching the debugger, the logic error was obvious.

Having spent the last couple years on our book The Practical Guide to Defect Prevention I also have a new appreciation for the simple typo and for the minor off-by-one errors that shift the UI by a pixel and stuff like that - they are only noticeable over time, but can turn out to be huge.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RS: It’s imperative to keep the big picture in mind. Many times, the test team can get so focused on perfection that they miss the obvious things that drive down the users' perception of quality. A large scale test effort might help an API work perfectly and handle error conditions gracefully, and performance is rock solid, but there’s a typo in the intro dialog or an edit box is truncated. Keep the user experience front and center. If a bug exists, but the customer doesn’t run into it, is it really a bug? :) Consider how to prevent the bug from appearing in the first place - invest in defect prevention techniques (such as root cause analysis, failure modeling, customer experience, etc.), partner with development, and monitor user feedback.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RS: It sounds a little odd - but I think testers can get in a box and try to be too thorough. The test matrix continues to explode and it is not usually possible to test everything. Again, step back to look at the big picture, and then assess the risk of missing defects in a certain code path. If a particular block of code hasn’t changed, then maybe it’s OK not to test that for a given release. The use of well known risk analysis, probability, and modeling techniques can help guide the test effort and improve effectiveness.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
RS: "Quality at the source" is a well known phrase that should apply in software development. Manufacturing and assembly line work moved many years ago to empower machine operators or assembly line workers with the ability to improve quality. The parallel for software development is improving the quality of the code as its written. Obviously, things like code reviews and inspections, design patterns, unit tests, etc. are a great start, but again, thinking about the big picture, how do we involve users earlier in the design and coding processes to prevent bugs before they are designed or coded into the software?

DDJ: Is there anything else you would like to say?
RS: Please check out The Practical Guide to Defect Prevention and www.defectprevention.org, and definitely send me any comments, feedback, or suggestions.

[See my Table Of Contents post for more details about this interview series.]

Matt understands that you may not agree with what he has to say, so he offers only two guarantees when you talk with him: that you'll leave the conversation thinking, and that you won't have been bored in the meantime.

Here is what Matt has to say:

They pushed the demo into production, then brought in the new hire for "maintenance." I was the new hire.

We had no requirements process, no change control process, no version control, no code review, nothing. Oh, and the software didn't work in the first place. All I knew was poke-testing, also known as happy-path testing.

The results, though tragic, were predictable - and painful. I thought to myself "There has got to be a better way" and started learning. Along the way, I found that I actually enjoy software testing; that this task many people think of as dull and boring, when done well, was anything but.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MH: The huge disparity between excellence and mediocrity. A few years ago, Brett Pettichord came up with his concept of "Four Schools of Software Testing", but I believe he missed the biggest one - the oblivious school. So many people in the field today have little knowledge about testing and no interest or incentive to learn. (No, I am not worried about offending them; they don't read blogs.)

I remember when Extreme Programming first became popular, and people like Ron Jeffries and Kent Beck were claiming that testing roles were ineffective and obsolete. I thought to myself "To think that, you must have never met an effective software tester" and that is probably true. They hadn't. The only interaction they had had were with members of the oblivious school.

It amazes me when people talk about test cases run in a day and they count single digits. Of course, that is possible - IF you spend four hours a day in meetings, and IF you create detailed test cases according to the IEEE 829 spec, if you use no automation, and if you use no exploratory or rapid methods.

When I run informal testing challenges at conferences, among my colleagues, if someone can't run ten test cases in three minutes, I'm surprised.

DDJ: What is the most interesting bug you have seen?
MH: The Microsoft Paint Image->Attributes->Height 1, Width -1 bug.

It's interesting because it's an unsigned to signed rollover (didn't those go out with thin ties in the early 1990's?), that it is in a mature product, and that it is probably not worth fixing. It's not worth fixing for a couple of reasons - First, Paint is a giveaway product that demos the features of windows. If you want something solid, get Photoshop or anything else. Second, there is a clear work around; -1 just plain isn't valid; the people that trip it are testers. And third, I hope, it's a great bug for testers when they do demos of exploratory testing.

In that case, I hope Microsoft keeps it around a long time.

DDJ: How would you describe your testing philosophy?
MH: Great Question. My short answer is that when testing is actually running the program under test ("dynamic testing"), you are going through a sort of investigative process of "will this software meet my customer's needs."

Unless the software is extremely simplistic, that turns out to be a rather complicated question to answer. We can never know for sure - there is always one more test to run. So "Are we done answering the question yet?" turns out to be another tough one. Then there's "What do we know with the research we have done so far?"

So I see testing as answering these really tough questions in a creative way. Any tool we can use to accomplish that - scripting languages, driving the browser, FIT, jUnit, Ruby, Windows Explorer, Task Manager, Excel Macros, VBA - any tool we can use - may be helpful.

I don't think you have to be a computer programmer to be an effective tester; I would hold critical thinking as the key skill. I've known many testers who didn't know the testing literature but were wildly effective - in every case, they were extremely strong critical thinkers. Having a CS degree just gives you the ability to do some things *really* *fast* and, in some cases, at all.

I generally split tests into two overlapping groups - developer facing tests and customer facing tests. The Dev tests are TDD and Automated Unit Tests - the Customer-Facing tests are what we now call "acceptance tests", but also "traditional" black box testing. For developer facing tests, I like the idea of a standard regression suite that makes sure we didn't break anything as we change our code. The Pavlovian response to the green bar - "oh, good, I green barred, I can go to sleep now" worries me in some cases however, because in that whole test "cycle", the person didn't do any critical introspective - any critical investigation - all the stuff I defined as testing above.

Also, I don't know if your readers are familiar with the mine field testing analogy, but I'm less and less in favor of running the exact same tests every time. Two common approaches that avoid that are model-based testing and exploratory testing. I kind of run down the middle; I use whatever automation tools that I find valuable in the moment to help me determine the state of the software under test.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MH: I think there are a lot of "red herrrings" out there about software testing. The first is probably the cult of requirements. Look, we've been saying we need better requirements for twenty years; that strikes me as placing testers in the position of the victim. "Get me better requirements" / "I need to be involved up front" / "I need a seat at the table" / "I need management buy in" / whatever. All of those are good things, but essentially, as an industry, we are insisting that other people change, not us. At a recent lunch conversation with Michael Bolton, he said that if he got involved early, that's great, but if not - he still wanted to do the best possible job with the time and resources he was given. I respect that stance.

We need to spend more time talking about how to improve our own skills, and less time asking other people to change to make our jobs easier.

Another one is the cult of documentation, which says that every test case must be documented in excruciating detail, then reviewed. One thing I know: If you spend eight hours a day documentin', one thing you ain't doin' is testing the software.

On the flip side, one thing I think we should *not* ignore, that *is* important, is writing skill (and, of course, testing skill, but I think I already mentioned that). Spending eight hours documenting is bad, but it's worse if you don't have the writing skill to make what you are reading compelling - in that case, nobody's going to read it. Or if you don't have the ability to sift the wheat from the chaff.

It's really easy to document the wrong thing. Then you didn't just waste your time - you've wasted the time of everyone who will ever read your document.

DDJ: Is there anything else you would like to say?
MH: Don't Belong, Never Join, Think For Yourself - Peace. No, wait, that's Victor Stone's Tag Line. Bummer.

I should add, probably to the Philosophy section:

It is so easy to misconstrue and miscommunicate things in the world of software testing. For example, I wrote that we answer questions like "Does the software work?"

Of course, we can't answer a question like that. Instead, on our best days, we can give a rough estimate of the state of the software under test. Likewise, we can't answer "Are we done?" - instead, we can provide information for decision makers, and let them decide whether to take one calculated risk (shipping buggy software) or, possibly, another (missing a market window).

That all depends on our mission - bug hunting vs. assessing quality are two very different missions. However, some missions are better than others. For example, the mission of "verify the software is correct", when the application has anything more than trivial bugs, is a pretty fruitless one. How can you verify a property that is not true?

[See my Table Of Contents post for more details about this interview series.]

These days Mike is Director of Development for a not-yet-unannounced product at Microsoft. Here is what Mike has to say:

DDJ: How would you describe your testing philosophy?
MZ: I’ll assume we are not talking about life critical software. I don’t buy that developers can’t test their own code. I do buy that tests, and developers (and frankly customers) train to correct paths and that diversity will uncover new bugs. I think developers should write correct code. And correct tests. And then run them. With automation. Then an independent engineering organization with a quality mandate should add mostly automated diversity to the mix. The combination of automation, tests and dashboard/reporting should be tuned over time to catch more and more bugs. When the code looks settled and usable in practice, a beta should be run. Not so soon that the feedback is noise due to ongoing code churn and not too late that feedback can’t be acted on. Concurrent with the correctness efforts, automation and tests that validate stability and performance under load should be developed and applied. When the product looks solid, it should be shipped, with telemetry to validate real world experience. Telemetry data should be collected and processed with automation. The entire machinery should be continuously improved.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
MZ: Systems are becoming more distributed and the business cycle is shortening. I don’t think my philosophy above works in this environment. Perhaps some kind of automated “sparse testing” is the answer.

[See my Table Of Contents post for more details about this interview series.]

Dunno yet whether I'll be attending. However, Michael Bolton will be. Matthew Heusser told me he is going. Pradeep Soundararajan is flying all the way from India to participate. Jerry Weinberg is keynoting. And that's only the tip of the iceberg!

While CAST will be a lot of adjectives, "boring" isn't one of them!

These days Eric works for Turner Broadcasting, where he tests the software that manages which commercials and programming content air when. Hey Eric, schedule me a Star Trek marathon this weekend, will you? <g/>

Here is what Eric has to say:

My approach is typically as follows but each test session depends on the context.

Select a chunk of AUT [application under test] functionality that devs consider testable, pose questions to business people until I am clear on how they expect the AUT to function, pose questions to devs until I am clear on how they intend the AUT to function. At this point, I usually have some bugs to log before even starting a test. Next, I attack said focused chunk of the AUT. This begins by helping the AUT to do what it feels most comfortable doing. Once it has proven itself to work consistently, I introduce variables (one at a time) that become more complex, until the bugs begin appearing. I let the tests generate from my mind as I witness behavior. I use the popcorn heuristic, and when the bugs begin to stop popping, I end my exploratory session. Finally, I break out my scripted tests, mark them as pass/fail accordingly, and add any valuable ones that were missing. These scripts are fragments, and only contain one step (i.e., do this, expect this). I’m a firm believer in James Bach’s mine field heuristic, and I tend to encourage different paths through the mind field via intentional vagueness.

I spend 20% of my time automating new tests (or fixing broken automated tests). Lately, these tend to focus in areas of performance testing popular paths through the UI and measuring time spans of UI experience and straight service calls. I use a home grown keyword-driven framework wrapped around QuickTest Pro. I’m also fascinated by monkey tests that randomly make decisions on what to do and can run indefinitely, though I only have a few of these and they aren’t good enough to find bugs.

Is that a philosophy?

DDJ: What is the most interesting bug you have seen?
EJ: Unfortunately, I saw this one in production...in an internal HR app I tested. After managers began complaining that performance appraisal comments they had written for their employees began disappearing, I decided to quiz the users and do more testing in said area. I quickly realized the employees could actually delete their manager’s hidden comments by saving new comments of their own. This was a small company and at the time we had no backups to restore. Most managers lost all their free form text for their performance appraisals and didn’t know it until they were about to conduct their face-to-face performance reviews. I was asked to send out an apology email to the entire company for missing that little bug. It was humiliating.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EJ: Though I haven’t read anyone else’s response, I’m sure I’m not alone on this one. Yes, writing manual test cases with details (e.g., UI stuff) is always emphasized and I think it is a major time waster. From my experiences, these test case details are typically ignored upon execution because they are wrong. Which begs the question, should I fix test case details and lose test time or leave them as is, which would leave a trail of incorrect information?

Another typically emphasized metric is the percentage of automated tests. My management loves to set goals like 50% of your test cases should be automated. These metrics are silly because they assume there is value in automating 50% of my tests, which there may or may not be. But I guess if I only write 10 tests, I only have to automate 5 of them. Hmmmm...

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
EJ: You should ask what frustrates me as a tester? I would answer, that there is no stopping point. Developers can sit back and admire a completed app, knowing their code can do what the specs say it should. They only need to provide one solution to a problem. Testers can never sit back and admire a completely tested app because they know if they have more time they can find more bugs. Testers are not providing a solution, we are inventing problems. Yet, as testers, we are constantly answering questions like “Are you done testing?”, “Can we ship?”. These questions are the stuff nightmares are made of.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EJ: As WPF, Silverlight, Flash, and other new UI technologies become more common, there will probably be a period where popular automation tools like QuickTest Pro and SilkTest lag behind in their support. My dev group decided not to use WPF last year, partly because QuickTest Pro did not support it and the folks at Mercury were not aware of any plans to support it.

On the flip side, it appears that Test Driven Development will continue gaining popularity for the next several years. This is good for testers! My dev team currently has over 3500 unit tests, which us testers walk through and use as build acceptance tests. It gives us the piece of mind that a huge chunk of the basic functionality is working. We spend less time chasing the boring environment config problems and basic functionality, and more time looking for the fun, less obvious bugs.

[See my Table Of Contents post for more details about this interview series.]

Here is what Evan has to say:

I was visiting with a family friend who was already a professional programmer. I was a Senior in High School and they were helping me wrestle with the question of whether I should major in Computer Science or Computer Engineering. I knew it’d be one of the two, but I really didn’t understand the difference. At some point, he left me to play with some budgeting software he had created for his dad to use. When he came back in the room I told him I found a bug in it. One of the values was coming up negative when it shouldn’t have. He asked if I knew why that was happening and I thought it was pretty obvious and quickly said "Yeah, I’m guessing you are using float instead of a double and I’m overflowing it." What I didn’t realize was that he wasn’t quizzing me, his mind was racing through mathematical and algorithmic errors he could have made but came up empty. It turned out that everywhere else, a float was fine – the values never got that large. But, the total had to be a double.

It was because of that that I landed an internship with his company the next summer where my job was to test desktop software to do mainframe session management. Without being asked, I found automation software to use, devised my own way to measure my "coverage" of the system, and left them with about 80 pages of documentation on the test methodology, instructions on how to run the tests, and a full gap analysis of what I didn’t cover as well as fully documented automation scripts.

They had thought I’d just test the software by using it (ad-hoc testing) and log bugs all summer long. I was under the assumption that clearly this being a professional environment "just" doing ad-hoc testing wouldn’t get me a return job the next summer so I had to step up my game, but I had no idea what that meant so I sort of made it up. Back then, I couldn’t just search the Internet for "software testing". I just thought – if I was in charge what would I have to see from a college intern to be impressed.

I don’t know if that is a statement about how bad they were or how good I was (the company made solid software, made a lot of money, and was bought out by one of the largest software firms for a crazy sum of money before the tech boom so I don’t think they were that bad), but it was a very positive experience for me because I ended up being somewhat self-taught in testing vs. being ramped up by others. By deriving a good test strategy on my own I could better appreciate what others had done in the field by the time I found articles and books on software testing. It’s sort of like learning calculus through memorization vs. derivation.

DDJ: What is the most interesting bug you have seen?
EG: One time I was doing memory leak testing on a streaming media system and found that, under the right conditions we would find an exponentially increasing memory leak happening, but at the same time a similarly decreasing CPU utilization pattern. Since all of us (dev, test, and PM) had inherited this giant mess of code with little documentation, none of us could figure out why we’de see a pattern like this – it just didn’t make sense. Luckily, I had proven myself so the typical route of questioning the tests didn’t happen :).

We set out to figure out what was going on and we found that there was some exponential backoff code that kicked in when the connectivity pattern was in a certain state. That explained the CPU graph, but only further made the memory leak graph unexplainable – you’d expect to see the opposite effect. Well, it turned out that memory wasn’t cleaned up in each case and the algorithm was trying "harder" each time, but waiting longer in between. "Harder" meant doing more work with more connection attempts that weren’t getting cleaned up.

None of this could have been found without collecting a ton of data and graphing them. A picture (created by automated tests), in this case, was worth 1,000 hours of manual testing!

DDJ: How would you describe your testing philosophy?
EG: There are two elements to my philosophy. The first is "Always have a balanced, but wide, test arsenal". If you over invest, you can miss stuff and not be well equipped with the right debugging and diagnostic tools when a certain bug comes around. You have to know when to use automation, know when to use manual testing, and know how to do manual testing efficiently and effectively with "directed ad-hoc" testing. The first thing I do when faced with testing a new component – no matter how big or small – is to create the big picture of what tools I need in my tool box to enable me to attack it from every angle, telling me everything about the system I need. These tools could be anything from software hooks, to well placed “printf’s” (asserts, exception handling, etc), to diagnostic tools, to linear functional regression tests. Once I have the big picture, the second piece kicks in – customer focused testing.

By customer-focused testing, I mean thinking about what sort of bugs each piece, or the depth of each piece, of the picture would find and what functionality it would verify. I prioritize based on the impact that bug would have on the customer. I’ve seen a lot of people, especially junior testers, get excited by finding a corner-case exception but ignore a layout issue as "fit and finish". In reality, the exception could be an extreme corner case – severe, but hardly ever hit – but the layout bug could chop off important text or cause extra mouse clicks in a usage flow that a user goes through 50 times a day. Likewise, looking at how actionable error messages are. Performance (or at least perceived performance) and other characteristics of the software are important to measure. This is how I prioritize the big picture I create and ensure that I build the test tools that will find the most important – from the customers point of view – issues. In other words, I really like to test with the end user in mind.

Once I have the prioritization, I go through and create all the cases as manual ones. That way, at the end of the project, I’ve at least got a record of all the testing I wanted to do and have a backup plan in my back pocket (using an army of manual testers to hit everything). That usually takes on the order of days. Then I start automating, building the tools, etc. Aside from a great backup-plan, this approach has a secondary benefit – you understand the testing you want to do with the automation. The best automation is born from manual processes that were repeated so often that they are well understood and could be replicated easily, without loosing potency, by code. I’m not saying you have to run each test case 100s of times before you can automate, but by at least laying out all the tests as manual ones, it requires you to think through the steps and run through them a few times to make sure they are repeatable and predictable. This small tweak – creating all the tests as manual ones before automating – has saved my teams time and time again either because we can deeply Plan B if some automation doesn’t happen to fit in the schedule or because we thought through the test cases first which saves us from going in the wrong direction and throwing away a lot of work. This is one of the highest return on investment activities you can do on a project.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EG: Testing the parts vs. testing the whole. As I said in my philosophy, you need to find the right balance. I’ve used products, as I’m sure everyone has, that are just dreadful in one way or another. If you talk to individual testers, they will claim their piece is only contributing to that metric in a positive way. However, the sum of the parts is much smaller than the characteristics of the whole. Performance is a great example of this. If you aren’t looking at both the product-wide and the "nuts and bolts" testing of all the pieces that make up the flow of that scenario, you won’t know what the user's experience will be, nor will you know where to look if that scenario’s performance doesn’t meet the bar. I’ve seen many large-scale projects where too much emphasis was put on the parts, but little ownership of the gaps between them existed. This, at least, has been in some of the projects I’ve been exposed to, and this isn’t a comment specific to Microsoft.

This isn’t a direct answer to the question, but I think too much emphasis is often placed on component level testing and I think some of that should be traded for a look at the whole to strike that balance. Using the second part of my philosophy, I’d prioritize against the severity and likelihood of a customer coming across a bug to know what to trade off.

It is admittedly a tricky business. Is a data loss bug that one person out of 10,000 will hit more important to go after than a more benign issue that will cause one out of 1,000 to have to reboot every 10th time they run the program? Is truncated text on a localized build with only 5,000 projected users more important than every users needing 2 more mouse clicks to do a common task more important? At the end of the day, the answer is going to lie in what is motivating and driving the business so the answer could be different in each company, in each product. But, you need to really understand those elements well. Not all bugs are created equal.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EG: We need to learn how to drive quality - in every sense of the word.

Software Engineering is unlike most other Engineering disciplines, in that the notion of "quality" extends far beyond the traditional bounds. Ease of use, doing what customers want and need from their software, easy and unobtrusive upgrades and patches, etc. are all necessary. You find these notions in other engineering disciplines, but few have all of them like software engineering.

I think our challenge is in learning from these other disciplines so it doesn’t take us 100+ years to become as refined as they are in their approach to quality assurance, yet still enable quick turnaround on new features, and driving quality across the entire product development cycle, not as a bolted-on activity that happens near the end. Again, this is not a reflection of Microsoft, but rather what I’ve seen on average across the industry.

[See my Table Of Contents post for more details about this interview series.]