THE BOOK OF TESTING

Thoughts From a Braidy Tester

April 2008

Bob Dewing has been managing testers for as long as I have known him, which means for at least the past ten years. He has been so focused on doing that well that I don't have much else to say about his testing career. So I will have to resort to telling anecdotes from when I worked for him. Well, maybe I shouldn't, because then he will likely turn bright red with embarrassment, like the time he was presenting at an all-hands meeting and his laptop went to screensaver...which his wife had customized to say "I love you Bob!"

There I go, telling stories after I said I wouldn't. I'll stop now and turn you over to Bob.

Here is what Bob has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
BD: I would love to have a story about a 'calling' to test, but at the time it was about preserving my mental state and helping the business. In the early 90's I worked in product support for a small privately held company that had no formalized testing process. In fact I witnessed the owner walking in to where the developers sat and yelling out 'who compiled last?', someone raised their hand and he hands them a disk and said 'great, copy it on here...we're shipping today'. Needless to say the product support role was a bit of a challenge and we ended up needing more and more time to research customer issues. The head of the department and I eventually convinced the owner that it would be cheaper (PSS was getting paid overtime to research bugs, near monthly product updates were needed, etc.) to create a formalized testing team. From there it was reading about manufacturing quality processes, talking to other companies about their processes, and coming up with an approach that 'made sense' to us. It felt like we were blazing a new trail and we were seeing immediate results and I was hooked.

DDJ: How would you describe your testing philosophy?
BD: I think the way I got into testing (through PSS) has stuck with me and defined my style or philosophy, and that is:

• Customer focused - regardless of where your feature is in the stack, you have some impact to the customer. It's super important to understand how your work is used and have some way to keep your finger on the pulse of your customer. This knowledge will not only help in the test case design stage, but also in the early feature design stage and can help you ask the 'how [or] does this solve the customer problem' question.


• Find the right tool/approach for the job - There's no single approach that will work for all problems. The typical personality profile of a tester is one that can look at a problem from different angles and ask 'what if' or 'what would happen if I...'. Learn about the different methodologies, even if your team doesn't use them currently. Look at how other testers with seemingly unrelated problems have solved them. You never know, with one small tweak that unrelated solution may all of a sudden become applicable.


• Continue to learn and improve the discipline - Related to above. I'm surprised at the number of experienced testers that don't continue their learning of the industry or discipline after they become comfortable with the basics of testing. Test can be an influential and dynamic discipline, continue to learn and evolve and bring your ideas to the table.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BD: Test has arguably undergone the most change of the primary engineering disciplines (dev/test/pm). Over the last 5 or so years more focus has been placed on how Test contributes to a company's success and that has really pushed the discipline, which is fantastic. We've come from being a feel good action ('did some testing...check') in a product release to being equal partners in the process, affecting product design and influencing engineering practices. There is still plenty of change on the horizon, so apart from keep pace, I see the challenges as being:

• Attracting, growing, and retaining great people - There is no longer a question of 'do we need to test', Test is critical to any software engineering process and now the challenge isn't how to establish a test organization, it is defining what a career in Test looks like. We can't just throw more entry level work at senior testers and expect them to feel like they have a fulfilling career or to picture what a 10+ year career in Test might look like. Find opportunities and challenges that are commensurate with the person's skillset, create mentoring opportunities, and share a clear career road map with individuals. This will help companies attract great people by articulating the role and career path in Test, as well as give their existing testers the ability to manage and drive their own careers and stay committed to the discipline.


• Finding ways to scale/managing the process - As systems become increasingly complex and the test matrix never seems to reduce, traditional defect detection approaches don't scale to meet the need. Test will need to better meet the need by (along with keeping quality top of mind throughout the organization) becoming more predictive in their scheduling or forecasting, moving towards static analysis tools that catch common defects, and creating re-usable systems that don't become obsolete version to version.


• Not focusing on a single metric or test approach (ie. automation) - A large number of passing automated tests in your lab doesn't mean you've achieved your goal of a stable, robust, secure, global-aware product that meets the customer's needs. Automation is super and can free the tester to think creatively and go deeper into the product. But it is easy for an organization to be crushed under the weight of their own automation and have it become more of a maintenance burden and each tester ends up spending more time with the overhead and never get to the interesting cases that the automation was supposed to free them up to do. There is no one datapoint to use as a quality indicator, don't become so focused on one that you ignore the others or so focused on one testing pathway that you ignore others (for example, automation vs. scenario based test sessions).

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
BD: You'll usually get one of a few different answers depending on which day you ask me....today it's....The stigma that the Test title still has - From the college campus trips to talking to experienced industry people, I'm surprised that a Test position is still seen as either poking keys at random and/or the place where developers that aren't good enough coders go.

DDJ: What is the most interesting bug you have seen?
BD: My favorite bug was back when I started, we got a call from a user whose cat walked across his keyboard and it locked up our application. Now who's going to test for that?!? But it's stuck with me and made me remember to think about the 'cat walking across the keyboard' scenario. Meaning, try to think about what could go wrong with your system and be leery of the 'oh, that could never happen' statement.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Paul Carvalho is one of the many Canadians who are actively contributing to the Test community. You may have read his article on internationalization testing for Better Software, his article on hiring testers for Red Canary, or his blog wherein he divulges the secrets of his testing success.

After work Paul's children help him decompress by ensuring he has neither time nor need to focus on any task for more than a few minutes at a time. Once his kids go to bed Paul studies origami and volunteers at Ontario's Perimeter Institute for Theoretical Physics. And still he finds time to review books and university courses, and to contribute to the Software Engineering Body of Knowledge. (Send kudos or complaints, depending on your view of that corpus, to Paul's email above. <g/>)

Here is what Paul has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
PC: I have thought about this question over the years, and I don't believe my first testing job was my first introduction to testing. Rather, there were several different experiences leading up to that first testing job that helped me realize when I had finally found my niche.

In High School, I used an Atari 800XL to connect to Bulletin Board Systems by modem. This was the 1980's and before the Internet was publicly available. I learnt a lot about communication protocols and could even whistle 9600 baud! (That means that I could dial a number and trick a modem into thinking that I was a computer just by whistling into the phone. What a geek. Ha, ha.)

Occasionally I would download some software, a game or an application of some kind without any documentation, and I would spend several days playing with it to see what it could do and what I could figure out about it. It was fun, a game: randomly and methodically hitting keys on the keyboard to see what happened, using tools and editors to read parts of the code, and making notes along the way to keep track of what I learnt. I rarely spent more than a week on any particular program before deleting it and moving onto the next one. This was similar to the Exploratory Testing that I do more formally today.

Later I went to university and majored in Physics. I studied and practiced the Scientific Method and problem solving on a regular basis. I became pretty good at modeling and pattern detection to help me "see" solutions in my mind to assigned problems. Essentially, I became very good at General Systems Thinking, a very useful testing skill, although I didn't know that name at the time.

While I was still in school, I held several contract positions as a programmer, and eventually came to work at Delrina in their Technical Support department in the early 90's. When I started there were fewer than 20 people in the department, but it was the right place at the right time and our department grew by four times within a 6-month period as the company experienced tremendous growth and popularity.

At some point, we started encountering more problems in Support than the QA team could help us solve, so we created our own test group within the Support department to help us get some answers quickly while the QA team focused on testing the next release. My thorough efforts in this Support Test group got the attention of the QA Manager. He came up to me one day and offered me a job in the QA team for the following summer, which I accepted.

On my next school break, I returned to Delrina in the QA department and worked on two cool projects: the modem compatibility project, and testing a new Forms-based information management system. In the four months I worked there I tested hardware (modems), managed a network of computers running through a series of automated regression tests, observed real honest-to-goodness Usability Testing, gained my first exposure to test techniques, and helped write automated test scripts to test the next generation Forms system.

So what did that leave me thinking about the act or concept of testing? I thought it was cool. I thought it was fun. I thought about the diversity in how testing was done in this one company and how each of the different ways helped to make a better quality product and experience for the end users. I reported bugs and they got fixed. I interacted with both Programmers and Tech Support people and enjoyed being the go-between. There was a "science" aspect to Testing as I drew upon my modeling and problem-solving skills from my school experiences. There was also an "art" to it as I drew upon my high school hacking hobby to feel my way through a system and find the information I needed.

I was bitten by the Testing bug and I liked it!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
PC: How complex it is to do a good job in testing. How diverse the range of skills are that you need to be successful. How important a role Psychology plays in software development. How many people vastly underestimate the effort required to do a good job of testing. How shockingly little most developers know about effective testing techniques. How little most educational institutions offer their students in the way of Testing instruction. How much the Testing body of knowledge still has to develop. How much debate there is over terminology and certifications. How hard it is to hire a good senior tester.

Testing has a solid basis in the Scientific Method and so it is open to many of the same theoretical, practical and philosophical debates that you encounter in Science. Things like: the nature of observation and bias, control and awareness of variables within your tests, the role of Statistics and Probability in test design and analysis, the Psychology of information presentation (i.e. for increased understanding), the variability in repetition of experiments, validity of assumptions, and so on.

The people who don't understand this generally do bad testing because their superficial view leads them to believe in simplified 'best practices.' Those of us who have a pretty good idea of what we're doing spend a lot of time trying to undo the damage caused by the processes, beliefs and expectations that sprout from these best practices.

I can't just pick one thing. Many things still surprise me.

DDJ: What is the most interesting bug you have seen?
PC: That's a tough one because I think many of them are interesting in their own way. I still have a very fresh attitude towards testing. Every day is a new opportunity to find an interesting bug. And I can't turn it off either. I see bugs everywhere - in the Grocery store, in newspapers and flyers, children's toys, restaurant menus, you name it. The world is a pretty buggy place.

Some bugs just appear to me by accident, some because I go looking for them, while others require lots of careful and detailed prep work and setup beforehand. For example, I once spent several days Performance Testing a web application to determine the maximum message throughput for a queuing component. I discovered that the application really supported 100 times fewer messages than expected. That was one of those "back to the drawing board" moments. Oops.

One of my favourite accidental bugs happened about a decade ago, when I worked for a company that made commercial math software. The software essentially had two parts: the math engine and the GUI. The company's corporate logo included an interesting multi-coloured 3-D graph representing a calculus function. I remember seeking out the mathematical function that produced the corporate logo and adding it to one of our regression test suites. In one development build, I discovered that the math engine had changed in a way that caused the math function to produce a different graph! I think the bug report I created had a title like "We can no longer produce our Corporate Logo with our own software!" They fixed it right away. We all had a good laugh over that one for many weeks afterwards.

One of the strangest bugs I remember happened when I was working in the Tech Support department at Delrina back in the early 90's. We were supporting Fax modem software in early versions of DOS and MS Windows. There was one customer who had bought our software and a supported fax modem but he just couldn't get the two to work together. My colleague who took the call tried everything. He checked and changed the computer's BIOS (this was before Plug-and-Play), added the necessary changes to the Windows 3.x environment, and tried a variety of modem initialization and configuration strings. Nothing worked. He felt bad but had to tell the customer that we just couldn't figure out why the software and hardware didn't work together. Some time later that same customer sent us a fax using our software with the message: "My computer got hit by a power surge during a lightning storm and the modem started working with your software! Thanks for your help!"

That fax was posted in our lunchroom for many, many months. That was funny. Weird too. Not a recommended fix though: modem doesn't work? Zap it with lightning! =D

DDJ: How would you describe your testing philosophy?
PC: I think a testing philosophy has several different aspects to it.

To begin with, I think that Testing is best understood in the context of the bigger picture of what is generally going on in Software Development projects. That is, software development is essentially the process of translating an idea from someone's head into computer instruction sets that perform some tasks useful for some purpose. Time and cost also factor in here, but people are the biggest part of the equation. So, if software development is a creative, people-based translation process, what is testing?

I believe that Testing is a way by which we empirically check the progress of the brain-to-code translation. If done well, Testing provides the feedback required to make the necessary adjustments along the way to ensure that we hit the target at the end of the project. If done poorly or not at all, the information may be insufficient or come too late to be useful in this process. Missing the target may have huge consequences. For example, people may lose their jobs, people may lose their finances, or people may die, to name a few.

How is testing done? It is simply done by shifting your perspective on the creative process, asking questions, making observations, and reporting findings to the decision makers responsible for the product release. I suppose "simply" may not be the best term here since each one of those steps require skill and knowledge in order for you to be effective and successful.

I believe that the Scientific Method and Statistics are cornerstones in effective testing approaches. I believe that you have to learn how to recognize a bug and it begins with how you think about the system and what you think the correct expected behaviour should be. Psychology also plays an important role in Testing. Testers are influenced everyday by many people and distracted by many things, and we need to be aware of the influences and biases that we bring to the job.

Because of the impossibility of complete testing, decision-making and good judgment skills are also important in Testing. Key project factors such as quality, time and cost need to be integral parts of the daily decision-making processes, whether you are trying to decide which features to test (or skip) or helping to decide when to ship the product. "Risk-Based Testing" is a popular approach that people may use to evaluate the importance of their testing effort based on the impact of an event and the probability of occurrence (back to Statistics here). To be useful, these decisions need to be made regularly and not just once per project.

Overall, I see testing as a creative, methodical, investigative, disciplined, fun, people-based way of helping to turn ideas into products. Testing is a service to the development organization, to the creative process of figuring out what should be in the product and how it should work. Good testers generate good, timely information for the stakeholders to use - in time for them to use it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PC: I've watched Agile Development methodologies slowly gain momentum over the last five years. I knew many development managers who were intrigued by the Agile Principles five years ago but were afraid to try the new ways of developing software. I have seen a shift happen in Development as more and more companies are adopting agile practices. That shift has yet to begin with Software Testing.

There's an approach called Exploratory Testing (ET) that has been around for a while but many people have been afraid to try it. Unfortunately, there appears to be much confusion over what ET really means and how to do it. As a result, there are many test teams still doing things the old way, relying upon their trusty test documentation and not realizing that they are making themselves obsolete.

This 'old way' that I am referring to is a documentation-based testing approach where someone creates documents with many, many test cases and then someone (else?) executes those test cases to generate test reports and other documentation. I suppose it keeps the Paper Industry prosperous.

One of the major problems I have with this approach is that it fundamentally follows a "Waterfall" model, and therefore is not compatible with Agile practices and principles. I have already seen some companies struggle to understand why their testers aren't really helping on Agile development projects. Therefore, the quick conclusion must be that testers aren't required on Agile projects, right? Not so!

Exploratory Testing, when performed by skilled testers, may be a very effective complement for Agile projects. It is adaptable, responsive to change, effective and produces exactly the kind of information that developers and stakeholders need on the project.

So, over the next five years, I see several big challenges facing us.

For testers:

1. Testers who wish to remain useful will need to learn new techniques and new skills that will enable them to be more effective, adaptable and responsive to change in a fast-moving industry.


2. Exploratory Testing is going to gain popularity in understanding and usage. This is going to change how test teams work together. It will also change how outside people work and interact with test teams. People who are afraid of change should just get out now. ;)

For the test discipline:

1. More courses in Software Testing and good testing practices in context need to be created and integrated into Computer Science and Software Engineering programs at educational institutions everywhere.


2. New Test Management and communication frameworks need to be developed to help test teams work effectively in a diversity of project sizes. There are very few options available today to help you manage or communicate your ET efforts. More research and collaboration will be required to help the test discipline move forward in a good direction.


3. Most of the vendor Test Management tools are based upon the central idea of a "test case." When that concept becomes obsolete, vendors will have to redesign their tools to be useful to testers again.

In summary, the biggest challenges facing testers are in self-development and embracing new approaches that improve their effectiveness and usefulness. The biggest challenges facing the test discipline are in educating more people in useful testing knowledge and techniques, and in developing new tools and techniques to help us manage the evolving testing practices. Shouldn't be too hard, right? =)

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

How might you know Johanna Rothman? Let me count the ways. She is the author of Hiring the Best Knowledge Workers, Techies & Nerds, which teaches manager types how to find people like you and me and convince us to work for them. She teamed up with Esther Derby on Behind Closed Doors: Secrets of Great Management, which helps manager types manage people like you me - and helps us understand our managers. And most recently she published Manage It! Your Guide to Modern, Pragmatic Project Management, which helps manager types - and people like you and me - keep our projects out of the Business Blow-Ups section of our local newspapers. If blogs are more your style, you can get Hiring Technical People and Managing Product Development that way as well.

Can you tell Johanna runs a management consulting business? <g/> Also why she co-hosts AYE and helps Jerry Weinberg teach PSL?

Johanna follows the Helpful Rule in everything she does, and her books, workshops, and consulting practice all help you understand how to best do the Good Job you want to do. Here is what Johanna has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JR: I became a tester when I went to work for a new company where I didn't know the products, didn't know the language, and didn't know how the products were applied (Symbolics, the lisp machine manufacturer). I wanted to work with the smart people who were there, and I thought Symbolics was poised for greatness. They were; it was brief greatness :-(

Since I'd been a developer for the previous 9 years, I applied what I knew about development to testing. I did these things, just as I had when I had been a developer:

• Explored the application for a short while to decide where to automate.


• Used combinatorial testing techniques.


• Learned quickly to ask about the vital few scenarios that our customers would use, so I could test those in detail, looking for the places our developers had blind spots.

I'm still not an expert in Lisp, but at one point, I was good enough to write lots of little pieces of code that we could use for tests.

What impressed the developers was how "nasty" I was. I found problems no one imagined could exist. I didn't think I was all that nasty; I was using the product the way I thought a user might :-) The most valuable thing the tester brings to his or her work is the curiosity and skepticism about how the product should/does work. Developers could never believe all I did was try to approach the application with an open mind. "Why did you do that?" "It seemed reasonable to me." "It's *not* reasonable!" And then, if the developers didn't fix the problem, sure enough, a customer *would* find exactly that problem. A number of developers were surprised by how creative testers needed to be.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JR: I am constantly amazed at how easy it is to break systems that provide an interface to the general public. For example, my local grocery store upgraded their software last Friday. On Sunday, when I did my grocery shopping, I whipped out my credit card to pay, and the cashier said, "Oh, if it's a <large-bank-name>, we can't take that today." "Why not?" "The debit part of it doesn't work. You need to use a credit card."

How is this possible? How could the vendor not test (or not realize there's a problem with) the software for a common banking partner? I don't know about the cost to my local grocery store, but you can bet I'll be checking the prices of what I buy. If the vendor can't get the payment details right, how will they get the cost details right? (You and I both suspect that the price lookup is different from the payment verification, but if their vendor got one thing that wrong, why wouldn't the vendor make other mistakes?)

I don't understand how producers can continually *not* test their most common scenarios. These common scenarios aren't just the happy paths; they're the common ways people interact with the system, whether that scenario is a common mistake or a typical job the user wants to accomplish. I'm surprised when testers (and developers) don't break those scenarios down from system level tests to smaller and smaller parts of tests. Knowing that your product works in its most common form gives me a feeling of confidence about the system. And, if I can't prove that the common scenarios work, I have areas I can address in more detail.

DDJ: How would you describe your testing philosophy?
JR: I want to make sure the testers are testing the areas that provide the most value to the customer and the areas where we have the most risk. That means the testers have to work with the requirements people to understand the value to the customer. And, the testers have to work with the developers to understand the areas of risk.

JR: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JR: There are two important things for testers to know: Testers need to know about the systems they are testing and how to be friends with their developers. They need to learn the common product scenarios; the implementation language(s) and how developers can be tripped up by the language; what's most valuable to the customers; how to demand automation hooks from developers; and how to use those hooks to develop small pieces of automation. Testers who ask these kinds of questions also tend to be people who use their previous knowledge of other systems and apply that knowledge to this system.

Testers need to be friends with developers, so they can eat lunch with their colleagues and ask, "What was challenging for you? What should I look at? What should I avoid?" I don't necessarily believe the "what to avoid"--I'll ask why so I can think of ways to test.

Developers need to learn how to test as they go, so they can learn how their code could break. Developers have to learn that they can't just do unit tests; they need to look at larger chunks. There's a huge gap between unit tests and system tests, and most of that gap is the developer's responsibility.

Testers, along with developers, need to continually re-examine their approaches and ask themselves, "Am I learning something new from this testing approach? If not, what do I need to?"

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JR: This whole business of measuring how many defects there are by severity. Oh please. Severity is irrelevant. If you have defects in the code, it's dirty. Seeing one defect means it's ok to have more defects. If testers showed the total remaining open number of defects by week, instead of anything else by week, the trend would have a lot more benefit to the project. Developers need to learn to clean up the code, and the best way to do that is to measure the total number of remaining open defects.

Severity and priority might be useful for organizing when and what to fix, but people start playing the defect promotion/demotion game as soon as they start counting by severity. That game allows people to put blinders on, and release broken software.

Now, it's always a business decision about when and what to release, but the people who make that decision need the data to make a good decision. They can't make good decisions without trend data.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JR: What should project managers know about testing?

Project managers need to know which testing should be done by which people, and if those people are actually performing those tests. A PM who waits for the testers to test the product (of course, in three weeks!) is an incompetent PM. Software is a design problem, not a manufacturing problem. You gotta test the design as you go, which means no matter what lifecycle you use, you have to start testing from the beginning.

Test driven development (TDD) is really about design, and good PMs should encourage their developers to use TDD. But even if the organization is wedded to waterfall, or if they think they need an architectural phase, the testers should know enough to participate in architectural reviews, and design reviews. Testers who bring curiosity and skepticism will help the developers create a much better product.

Sometimes, asking the "dumb" questions help people realize what's going on in the product. I once asked "How does that queue work? I understand how things get into it, but I don't understand how things get out." Turns out, there was a design problem in removing things from the queue. If I hadn't learned enough about systems to know that, or if I had been uninterested in the code, I would never have asked that question.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Adam White manages Test Engineering and Escalations for PlateSpin. He makes the most of this dual role by using the customer problems he encounters to drive changes in his team's testing process and thus reduce the need for future escalations. As Test Manager part of Adam's job entails engaging his testers in philosophical discussions about testing and living his belief that teaching testing is more about helping people improve their ability to think about software than it is about instructing them in any particular technique or methodology. He also tinkers with neural networks and adaptive systems and such as he experiments with processes for analyzing financial markets and otherwise managing risks of various sorts.

Here is what Adam has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AKW: I remember working for a young start-up company near release and we were still finding bugs rapidly. Ship time arrived and the business side decided to ship the software. I was actually told "Stop testing - If you keep testing we will keep finding bugs. We can live with the bugs you have found. We have to ship the software". I realized that in commercial business-to-business software it's about the company's ability to manage the confidence and risk of the customer, not about finding every bug.

DDJ: What is the most interesting bug you have seen?
AKW: Do I have to pick just one? :) A scheduling dialog exhibited some interesting behavior found by Aqiqul Hoda, a tester on my team. The dialog contained a date/time control. If you had the date/time dialog open when the computer time changed to be on the hour an unhandled exception would appear. Aqiqul logged a bug for the unhandled exception without knowing the root cause. All of us thought he was crazy - until he reproduced it later in the afternoon. Eventually we did enough diagnosing and troubleshooting to get to the root cause. It was an amazing bug and great team troubleshooting.

DDJ: How would you describe your testing philosophy?
AKW: My testing philosophy used to be about stopping the shipping of bad software (bad in my opinion). Now my philosophy is focused on managing the confidence and risk of project stakeholders. My primary aim is to provide stakeholders with accurate and timely information so the project team can make informed business decisions about the software.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AKW: One of the most important things for me has been keeping my skills sharp. I'd like people to remember to practice the testing craft not only at work but in everyday life. Let testing permeate your life. Always try to look for new and interesting ways to do things.

I do this by reading a lot and testing random programs that I know nothing about, just for the sake of learning and keeping my skills sharp. I have gotten involved in testing communities and attended dinner meetings and conferences. I also do things that put me outside my comfort zone like improv classes. All of these things link back to my work.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AKW: The biggest challenge I see for testers is finding ways to test smarter. Systems are getting complex with technology, scenarios and interactions. This creates a huge testing space. It's no longer sufficient (and in some contexts never was acceptable) to bang away at the keyboard and hope "bad things happen".

The challenge for the testing discipline is getting fresh, young blood to pursue testing as a career. The testing world tends to get ignored in university curriculums and isn't even considered as a career option by most university graduates.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Joshua Williams  has been a tester throughout his twelve-plus years at Microsoft. That time has seen him working on globalization, testing drivers, and leading the USB support test process, on more architectures and versions of Windows than you probably care to hear about. He has worked with and on various automation frameworks and process improvement exercises as well, which experience is one reason he coauthored the recently published A Practical Guide to Defect Prevention. (See my review elsewhere on DDJ.) His primary focus nowadays is finding ways to help testers better enjoy their work, a mission I wholeheartedly support!

Here is what Joshua has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JW: I had been working in the back room of computer stores for a few years as the "fix-it guy". When I did my first interview loop to start at Microsoft, a test lead asked me to do some tests on an application he then loaded up for me. I am sure I didn't do it all correctly, but I found some problems, and passed the interview. I started by testing hardware compatibility. It was a conceptual leap for me to move from "making the machine work" and "pleasing the customer" to a testing mindset. Obviously pleasing the customer is still important, but in testing we strive to find, understand and log problems, not just find workarounds so we can get the customer up and running. It's the discipline of digging into the problem and getting it fixed that was the change. And it turns out, it's rewarding, too, just in a different way.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JW: After all the technical work, the code, the bug reports, etc., I would have to say that effective communication can make or break your career. I have made plenty of mistakes, and most of them were trying to get the right message across. Communication with your counterpart developers, the program managers, your lead or your employees can create trust, or it can erode it. Sometimes the unintended message is not just a result of what you say, but of how you say it. There are a wide range of strategies people take up when they communication with others. Some people get aggressive, some grow more passive. Your goal is to deliver the message you have in a way that you accomplish your design, not just to say the words. This matters in all kinds of scenarios from defect reports all the way to defending a bug before the release management team. It especially matters when leading a team. Communication can be tough to master, because it isn't a hard skill, but it often carries more weight than your technical know-how.

DDJ: What is the most interesting bug you have seen?
JW: There was this bug that only occurred because of a mix of power management and removable storage. It was interesting to me because it demonstrated how even good test teams can wear blinders. I am confident that the removable storage team did a full battery of tests, probably achieving incredible code coverage results with their testing of removable storage. The same is true of the power management team. But, they never got together. It isn't anyone's fault, but then again, it's both their faults. Should storage drivers and functionality include power management testing? Should the power management test team have a huge array of different kinds of devices and removable media? Where does the responsibility of one team end and the other begin? There aren't perfect answers to these questions. This kind of challenge is part of what makes testing as a career fun: it's not all black and white. To be effective, we have to look outside our boxes, and see the world a little more like the customer does. Only by doing so can we understand how to best test.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JW: Some of the best testing I have ever been a part of was a single SDET and a single SDE focusing on a specific feature in a tightly-coupled feedback loop. They made the most progress because they both shared the same goals: make the feature and the tests great. They both contributed to both goals, even though they owned only one each. This was great because they could communicate openly and honestly. They put away their egos and worked together. The "Quality" wasn't left to the SDET; they both owned it. Not every project will work out like that; but the core principle doesn't change: put away your ego and remember that everyone is responsible for the quality of the product.

DDJ: Is there anything else you would like to say?
JW: People need to read. Testers need to study other disciplines and understand what other companies are doing. I am lucky to be in a group of avid readers, and it has really pushed me to read more. I have really enjoyed reading books on business theory and process management from industries outside software. It has been remarkably educational and eye-opening in how I view quality. I would challenge testers to read. And, they can start with a book that I contributed to: A Practical Guide to Defect Prevention, with more info at www.defectprevention.org. It has a lot of info about software development, but also has a lot of info about things we have learned from other industries.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |