Testing & Debugging Blog 2008-05-06T18:50:52Z tag:,2008:/50 Movable Type Copyright (c) 2008, mhunter Five Questions With Elfriede Dustin 2008-05-06T18:50:52Z 2008-05-06T12:30:00Z tag:,2008:/50.33313 2008-05-06T12:30:00Z Elfriede Dustin has written numerous books and articles about testing and test automation. While I haven't read any of them myself, multiple someones evidently find her books useful as they have been translated into multiple languages and sell worldwide. She... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Elfriede Dustin has written numerous books and articles about testing and test automation. While I haven't read any of them myself, multiple someones evidently find her books useful as they have been translated into multiple languages and sell worldwide. She co-chairs the annual VERIFY conference, where one of the three presentation tracks focuses on (surprise!) test automation. She got her start in computers doing a different type of automation - using Wang computers to automate claims processing for the United States federal government. These days she is helping the US military automate their software testing, which seems a rather different challenge than the ones on which I work.


Here is what Elfriede has to say:

]]> DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
ED: Most surprising to me is that still often over 60% of the software development lifecycle is spent on software testing. For example, a most recent quote in Redmond Developer News states that "Industry analysts estimate that developers spend only about 20 percent of their time designing and coding ... the bulk of their time is spent on resolving application problems…" I have worked in various software development/testing environments, i.e. commercial and Government, and the testing challenges exist across the board, independent of the environment. Software testing techniques lag behind hardware testing techniques, for example, where use of tools is now regarded as mandatory. Today manual test generation for hardware is inconceivable; yet software testing often is still done manually in many companies. Software testing needs to advance to the same level as hardware testing; i.e. all software testing should be automated to allow for various efficiencies, such as software test reuse and repeatability.

DDJ: How would you describe your testing philosophy?
ED: My testing philosophy is to automate, automate, and automate. Manual testing is labor intensive, increasingly repetitive and thus error prone. It is archaic and cannot keep up with the software development advances. Despite the advances in development practices and tools, the goals of accelerating the rate at which systems can be delivered and reducing their costs cannot be met by simply writing software faster without comparable improvement in the practices and tools for testing the software. Software testing techniques are not keeping pace with software development advances. With the increased use of advanced software development tools, languages, and model driven architectures by software providers, software testing is rapidly becoming the "very longest and most expensive pole in the tent" when it comes to fielding new capabilities into production or to the masses. Automated testing is the answer. Automated testing tools and methodologies (such as the Automated Testing Lifecycle Methodology (ATLM) described in my book Automated Software Testing) have been developed and continue to emerge. Automated software quality tools growth is primarily due to businesses’ desire for higher quality, increased productivity, and faster time to market. Open source available automated testing tools are improving and STAF/STAX is one great example of a powerful open source automation framework.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ED: The most important thing for testers (or anyone) to know is that testing requires skill and expertise. Software testing is a profession and should not be considered only a stepping stone into another department, for example into development. Often the misconception exists that anyone who is able to hold a clipboard can be a tester. I have written various books and articles on the skills required to be a tester (for a list of my books, see http://www.effectivesoftwaretesting.com/Books.aspx); i.e. to be an efficient software tester the person really has to enjoy testing, and not only see it as a temporary involuntary assignment; have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, to understand the technical intricacies of the application-under-test in order to be able to automate the testing efforts, b) understand the business, and c) never stop learning and improving on the testing efforts. Developers need to unit test their code and make automated unit testing part of their build processes, which results in the most successful development efforts.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ED: The biggest challenge will be that testers and the test discipline will continue to play "catch-up" with development advances. With today’s processor cores tearing through billions of instructions per second and software builds of n-tier applications consisting of thousands of source files, computer hardware and software technologies have made many significant advances. Advances in development practices, methodologies, and tools which have been enablers in fielding phenomenal new, innovative, affordable, and timely information processing capabilities will continue to make headlines. In this existing paradigm of rapid development and edge of technology advancements, with a focus on reducing development time and creating re-usable capabilities, efficient and thorough testing of the software will be more critical than ever. Testing techniques and technologies will need to be able to keep up with the phenomenal advancements: Automated Software testing and self-testable components are part of the answer to this challenge.

DDJ: What else should I ask you?
ED: Where do you currently work and what excites you about your current job?

DDJ: What is your answer?
ED: I currently work at Innovative Defense Technologies. The exciting part about this company is that our goal is to improve software testing efforts across DOD and commercial markets, for example we are bringing automated testing solutions to mission critical systems at the Navy and soon to other parts of the DOD. IDT as a company is solely dedicated to improving software testing efforts across the board. We are working with edge of technology testing techniques and have some of the brightest leadership and developers assigned to the software automation tasks and efforts. IDT will change how testing gets done in the long run.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Bob Dewing 2008-04-29T20:54:24Z 2008-04-29T12:30:00Z tag:,2008:/50.33153 2008-04-29T12:30:00Z Bob Dewing has been managing testers for as long as I have known him, which means for at least the past ten years. He has been so focused on doing that well that I don't have much else to say... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Bob Dewing has been managing testers for as long as I have known him, which means for at least the past ten years. He has been so focused on doing that well that I don't have much else to say about his testing career. So I will have to resort to telling anecdotes from when I worked for him. Well, maybe I shouldn't, because then he will likely turn bright red with embarrassment, like the time he was presenting at an all-hands meeting and his laptop went to screensaver...which his wife had customized to say "I love you Bob!"

There I go, telling stories after I said I wouldn't. I'll stop now and turn you over to Bob.

Here is what Bob has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
BD: I would love to have a story about a 'calling' to test, but at the time it was about preserving my mental state and helping the business. In the early 90's I worked in product support for a small privately held company that had no formalized testing process. In fact I witnessed the owner walking in to where the developers sat and yelling out 'who compiled last?', someone raised their hand and he hands them a disk and said 'great, copy it on here...we're shipping today'. Needless to say the product support role was a bit of a challenge and we ended up needing more and more time to research customer issues. The head of the department and I eventually convinced the owner that it would be cheaper (PSS was getting paid overtime to research bugs, near monthly product updates were needed, etc.) to create a formalized testing team. From there it was reading about manufacturing quality processes, talking to other companies about their processes, and coming up with an approach that 'made sense' to us. It felt like we were blazing a new trail and we were seeing immediate results and I was hooked.

DDJ: How would you describe your testing philosophy?
BD: I think the way I got into testing (through PSS) has stuck with me and defined my style or philosophy, and that is:

  • Customer focused - regardless of where your feature is in the stack, you have some impact to the customer. It's super important to understand how your work is used and have some way to keep your finger on the pulse of your customer. This knowledge will not only help in the test case design stage, but also in the early feature design stage and can help you ask the 'how [or] does this solve the customer problem' question.

  • Find the right tool/approach for the job - There's no single approach that will work for all problems. The typical personality profile of a tester is one that can look at a problem from different angles and ask 'what if' or 'what would happen if I...'. Learn about the different methodologies, even if your team doesn't use them currently. Look at how other testers with seemingly unrelated problems have solved them. You never know, with one small tweak that unrelated solution may all of a sudden become applicable.

  • Continue to learn and improve the discipline - Related to above. I'm surprised at the number of experienced testers that don't continue their learning of the industry or discipline after they become comfortable with the basics of testing. Test can be an influential and dynamic discipline, continue to learn and evolve and bring your ideas to the table.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BD: Test has arguably undergone the most change of the primary engineering disciplines (dev/test/pm). Over the last 5 or so years more focus has been placed on how Test contributes to a company's success and that has really pushed the discipline, which is fantastic. We've come from being a feel good action ('did some testing...check') in a product release to being equal partners in the process, affecting product design and influencing engineering practices. There is still plenty of change on the horizon, so apart from keep pace, I see the challenges as being:

  • Attracting, growing, and retaining great people - There is no longer a question of 'do we need to test', Test is critical to any software engineering process and now the challenge isn't how to establish a test organization, it is defining what a career in Test looks like. We can't just throw more entry level work at senior testers and expect them to feel like they have a fulfilling career or to picture what a 10+ year career in Test might look like. Find opportunities and challenges that are commensurate with the person's skillset, create mentoring opportunities, and share a clear career road map with individuals. This will help companies attract great people by articulating the role and career path in Test, as well as give their existing testers the ability to manage and drive their own careers and stay committed to the discipline.

  • Finding ways to scale/managing the process - As systems become increasingly complex and the test matrix never seems to reduce, traditional defect detection approaches don't scale to meet the need. Test will need to better meet the need by (along with keeping quality top of mind throughout the organization) becoming more predictive in their scheduling or forecasting, moving towards static analysis tools that catch common defects, and creating re-usable systems that don't become obsolete version to version.

  • Not focusing on a single metric or test approach (ie. automation) - A large number of passing automated tests in your lab doesn't mean you've achieved your goal of a stable, robust, secure, global-aware product that meets the customer's needs. Automation is super and can free the tester to think creatively and go deeper into the product. But it is easy for an organization to be crushed under the weight of their own automation and have it become more of a maintenance burden and each tester ends up spending more time with the overhead and never get to the interesting cases that the automation was supposed to free them up to do. There is no one datapoint to use as a quality indicator, don't become so focused on one that you ignore the others or so focused on one testing pathway that you ignore others (for example, automation vs. scenario based test sessions).

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
BD: You'll usually get one of a few different answers depending on which day you ask me....today it's....The stigma that the Test title still has - From the college campus trips to talking to experienced industry people, I'm surprised that a Test position is still seen as either poking keys at random and/or the place where developers that aren't good enough coders go.

DDJ: What is the most interesting bug you have seen?
BD: My favorite bug was back when I started, we got a call from a user whose cat walked across his keyboard and it locked up our application. Now who's going to test for that?!? But it's stuck with me and made me remember to think about the 'cat walking across the keyboard' scenario. Meaning, try to think about what could go wrong with your system and be leery of the 'oh, that could never happen' statement.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Paul Carvalho 2008-04-22T19:12:21Z 2008-04-22T12:30:00Z tag:,2008:/50.32989 2008-04-22T12:30:00Z Paul Carvalho is one of the many Canadians who are actively contributing to the Test community. You may have read his article on internationalization testing for Better Software, his article on hiring testers for Red Canary, or his blog wherein... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Paul Carvalho is one of the many Canadians who are actively contributing to the Test community. You may have read his article on internationalization testing for Better Software, his article on hiring testers for Red Canary, or his blog wherein he divulges the secrets of his testing success.

After work Paul's children help him decompress by ensuring he has neither time nor need to focus on any task for more than a few minutes at a time. Once his kids go to bed Paul studies origami and volunteers at Ontario's Perimeter Institute for Theoretical Physics. And still he finds time to review books and university courses, and to contribute to the Software Engineering Body of Knowledge. (Send kudos or complaints, depending on your view of that corpus, to Paul's email above. <g/>)

Here is what Paul has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
PC: I have thought about this question over the years, and I don't believe my first testing job was my first introduction to testing. Rather, there were several different experiences leading up to that first testing job that helped me realize when I had finally found my niche.

In High School, I used an Atari 800XL to connect to Bulletin Board Systems by modem. This was the 1980's and before the Internet was publicly available. I learnt a lot about communication protocols and could even whistle 9600 baud! (That means that I could dial a number and trick a modem into thinking that I was a computer just by whistling into the phone. What a geek. Ha, ha.)

Occasionally I would download some software, a game or an application of some kind without any documentation, and I would spend several days playing with it to see what it could do and what I could figure out about it. It was fun, a game: randomly and methodically hitting keys on the keyboard to see what happened, using tools and editors to read parts of the code, and making notes along the way to keep track of what I learnt. I rarely spent more than a week on any particular program before deleting it and moving onto the next one. This was similar to the Exploratory Testing that I do more formally today.

Later I went to university and majored in Physics. I studied and practiced the Scientific Method and problem solving on a regular basis. I became pretty good at modeling and pattern detection to help me "see" solutions in my mind to assigned problems. Essentially, I became very good at General Systems Thinking, a very useful testing skill, although I didn't know that name at the time.

While I was still in school, I held several contract positions as a programmer, and eventually came to work at Delrina in their Technical Support department in the early 90's. When I started there were fewer than 20 people in the department, but it was the right place at the right time and our department grew by four times within a 6-month period as the company experienced tremendous growth and popularity.

At some point, we started encountering more problems in Support than the QA team could help us solve, so we created our own test group within the Support department to help us get some answers quickly while the QA team focused on testing the next release. My thorough efforts in this Support Test group got the attention of the QA Manager. He came up to me one day and offered me a job in the QA team for the following summer, which I accepted.

On my next school break, I returned to Delrina in the QA department and worked on two cool projects: the modem compatibility project, and testing a new Forms-based information management system. In the four months I worked there I tested hardware (modems), managed a network of computers running through a series of automated regression tests, observed real honest-to-goodness Usability Testing, gained my first exposure to test techniques, and helped write automated test scripts to test the next generation Forms system.

So what did that leave me thinking about the act or concept of testing? I thought it was cool. I thought it was fun. I thought about the diversity in how testing was done in this one company and how each of the different ways helped to make a better quality product and experience for the end users. I reported bugs and they got fixed. I interacted with both Programmers and Tech Support people and enjoyed being the go-between. There was a "science" aspect to Testing as I drew upon my modeling and problem-solving skills from my school experiences. There was also an "art" to it as I drew upon my high school hacking hobby to feel my way through a system and find the information I needed.

I was bitten by the Testing bug and I liked it!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
PC: How complex it is to do a good job in testing. How diverse the range of skills are that you need to be successful. How important a role Psychology plays in software development. How many people vastly underestimate the effort required to do a good job of testing. How shockingly little most developers know about effective testing techniques. How little most educational institutions offer their students in the way of Testing instruction. How much the Testing body of knowledge still has to develop. How much debate there is over terminology and certifications. How hard it is to hire a good senior tester.

Testing has a solid basis in the Scientific Method and so it is open to many of the same theoretical, practical and philosophical debates that you encounter in Science. Things like: the nature of observation and bias, control and awareness of variables within your tests, the role of Statistics and Probability in test design and analysis, the Psychology of information presentation (i.e. for increased understanding), the variability in repetition of experiments, validity of assumptions, and so on.

The people who don't understand this generally do bad testing because their superficial view leads them to believe in simplified 'best practices.' Those of us who have a pretty good idea of what we're doing spend a lot of time trying to undo the damage caused by the processes, beliefs and expectations that sprout from these best practices.

I can't just pick one thing. Many things still surprise me.

DDJ: What is the most interesting bug you have seen?
PC: That's a tough one because I think many of them are interesting in their own way. I still have a very fresh attitude towards testing. Every day is a new opportunity to find an interesting bug. And I can't turn it off either. I see bugs everywhere - in the Grocery store, in newspapers and flyers, children's toys, restaurant menus, you name it. The world is a pretty buggy place.

Some bugs just appear to me by accident, some because I go looking for them, while others require lots of careful and detailed prep work and setup beforehand. For example, I once spent several days Performance Testing a web application to determine the maximum message throughput for a queuing component. I discovered that the application really supported 100 times fewer messages than expected. That was one of those "back to the drawing board" moments. Oops.

One of my favourite accidental bugs happened about a decade ago, when I worked for a company that made commercial math software. The software essentially had two parts: the math engine and the GUI. The company's corporate logo included an interesting multi-coloured 3-D graph representing a calculus function. I remember seeking out the mathematical function that produced the corporate logo and adding it to one of our regression test suites. In one development build, I discovered that the math engine had changed in a way that caused the math function to produce a different graph! I think the bug report I created had a title like "We can no longer produce our Corporate Logo with our own software!" They fixed it right away. We all had a good laugh over that one for many weeks afterwards.

One of the strangest bugs I remember happened when I was working in the Tech Support department at Delrina back in the early 90's. We were supporting Fax modem software in early versions of DOS and MS Windows. There was one customer who had bought our software and a supported fax modem but he just couldn't get the two to work together. My colleague who took the call tried everything. He checked and changed the computer's BIOS (this was before Plug-and-Play), added the necessary changes to the Windows 3.x environment, and tried a variety of modem initialization and configuration strings. Nothing worked. He felt bad but had to tell the customer that we just couldn't figure out why the software and hardware didn't work together. Some time later that same customer sent us a fax using our software with the message: "My computer got hit by a power surge during a lightning storm and the modem started working with your software! Thanks for your help!"

That fax was posted in our lunchroom for many, many months. That was funny. Weird too. Not a recommended fix though: modem doesn't work? Zap it with lightning! =D

DDJ: How would you describe your testing philosophy?
PC: I think a testing philosophy has several different aspects to it.

To begin with, I think that Testing is best understood in the context of the bigger picture of what is generally going on in Software Development projects. That is, software development is essentially the process of translating an idea from someone's head into computer instruction sets that perform some tasks useful for some purpose. Time and cost also factor in here, but people are the biggest part of the equation. So, if software development is a creative, people-based translation process, what is testing?

I believe that Testing is a way by which we empirically check the progress of the brain-to-code translation. If done well, Testing provides the feedback required to make the necessary adjustments along the way to ensure that we hit the target at the end of the project. If done poorly or not at all, the information may be insufficient or come too late to be useful in this process. Missing the target may have huge consequences. For example, people may lose their jobs, people may lose their finances, or people may die, to name a few.

How is testing done? It is simply done by shifting your perspective on the creative process, asking questions, making observations, and reporting findings to the decision makers responsible for the product release. I suppose "simply" may not be the best term here since each one of those steps require skill and knowledge in order for you to be effective and successful.

I believe that the Scientific Method and Statistics are cornerstones in effective testing approaches. I believe that you have to learn how to recognize a bug and it begins with how you think about the system and what you think the correct expected behaviour should be. Psychology also plays an important role in Testing. Testers are influenced everyday by many people and distracted by many things, and we need to be aware of the influences and biases that we bring to the job.

Because of the impossibility of complete testing, decision-making and good judgment skills are also important in Testing. Key project factors such as quality, time and cost need to be integral parts of the daily decision-making processes, whether you are trying to decide which features to test (or skip) or helping to decide when to ship the product. "Risk-Based Testing" is a popular approach that people may use to evaluate the importance of their testing effort based on the impact of an event and the probability of occurrence (back to Statistics here). To be useful, these decisions need to be made regularly and not just once per project.

Overall, I see testing as a creative, methodical, investigative, disciplined, fun, people-based way of helping to turn ideas into products. Testing is a service to the development organization, to the creative process of figuring out what should be in the product and how it should work. Good testers generate good, timely information for the stakeholders to use - in time for them to use it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PC: I've watched Agile Development methodologies slowly gain momentum over the last five years. I knew many development managers who were intrigued by the Agile Principles five years ago but were afraid to try the new ways of developing software. I have seen a shift happen in Development as more and more companies are adopting agile practices. That shift has yet to begin with Software Testing.

There's an approach called Exploratory Testing (ET) that has been around for a while but many people have been afraid to try it. Unfortunately, there appears to be much confusion over what ET really means and how to do it. As a result, there are many test teams still doing things the old way, relying upon their trusty test documentation and not realizing that they are making themselves obsolete.

This 'old way' that I am referring to is a documentation-based testing approach where someone creates documents with many, many test cases and then someone (else?) executes those test cases to generate test reports and other documentation. I suppose it keeps the Paper Industry prosperous.

One of the major problems I have with this approach is that it fundamentally follows a "Waterfall" model, and therefore is not compatible with Agile practices and principles. I have already seen some companies struggle to understand why their testers aren't really helping on Agile development projects. Therefore, the quick conclusion must be that testers aren't required on Agile projects, right? Not so!

Exploratory Testing, when performed by skilled testers, may be a very effective complement for Agile projects. It is adaptable, responsive to change, effective and produces exactly the kind of information that developers and stakeholders need on the project.

So, over the next five years, I see several big challenges facing us.

For testers:

  1. Testers who wish to remain useful will need to learn new techniques and new skills that will enable them to be more effective, adaptable and responsive to change in a fast-moving industry.

  2. Exploratory Testing is going to gain popularity in understanding and usage. This is going to change how test teams work together. It will also change how outside people work and interact with test teams. People who are afraid of change should just get out now. ;)

For the test discipline:

  1. More courses in Software Testing and good testing practices in context need to be created and integrated into Computer Science and Software Engineering programs at educational institutions everywhere.

  2. New Test Management and communication frameworks need to be developed to help test teams work effectively in a diversity of project sizes. There are very few options available today to help you manage or communicate your ET efforts. More research and collaboration will be required to help the test discipline move forward in a good direction.

  3. Most of the vendor Test Management tools are based upon the central idea of a "test case." When that concept becomes obsolete, vendors will have to redesign their tools to be useful to testers again.

In summary, the biggest challenges facing testers are in self-development and embracing new approaches that improve their effectiveness and usefulness. The biggest challenges facing the test discipline are in educating more people in useful testing knowledge and techniques, and in developing new tools and techniques to help us manage the evolving testing practices. Shouldn't be too hard, right? =)


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Johanna Rothman 2008-04-15T23:10:56Z 2008-04-15T12:30:00Z tag:,2008:/50.32613 2008-04-15T12:30:00Z mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog How might you know Johanna Rothman? Let me count the ways. She is the author of Hiring the Best Knowledge Workers, Techies & Nerds, which teaches manager types how to find people like you and me and convince us to work for them. She teamed up with Esther Derby on Behind Closed Doors: Secrets of Great Management, which helps manager types manage people like you me - and helps us understand our managers. And most recently she published Manage It! Your Guide to Modern, Pragmatic Project Management, which helps manager types - and people like you and me - keep our projects out of the Business Blow-Ups section of our local newspapers. If blogs are more your style, you can get Hiring Technical People and Managing Product Development that way as well.

Can you tell Johanna runs a management consulting business? <g/> Also why she co-hosts AYE and helps Jerry Weinberg teach PSL?

Johanna follows the Helpful Rule in everything she does, and her books, workshops, and consulting practice all help you understand how to best do the Good Job you want to do. Here is what Johanna has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JR: I became a tester when I went to work for a new company where I didn't know the products, didn't know the language, and didn't know how the products were applied (Symbolics, the lisp machine manufacturer). I wanted to work with the smart people who were there, and I thought Symbolics was poised for greatness. They were; it was brief greatness :-(

Since I'd been a developer for the previous 9 years, I applied what I knew about development to testing. I did these things, just as I had when I had been a developer:

  • Explored the application for a short while to decide where to automate.

  • Used combinatorial testing techniques.

  • Learned quickly to ask about the vital few scenarios that our customers would use, so I could test those in detail, looking for the places our developers had blind spots.

I'm still not an expert in Lisp, but at one point, I was good enough to write lots of little pieces of code that we could use for tests.

What impressed the developers was how "nasty" I was. I found problems no one imagined could exist. I didn't think I was all that nasty; I was using the product the way I thought a user might :-) The most valuable thing the tester brings to his or her work is the curiosity and skepticism about how the product should/does work. Developers could never believe all I did was try to approach the application with an open mind. "Why did you do that?" "It seemed reasonable to me." "It's *not* reasonable!" And then, if the developers didn't fix the problem, sure enough, a customer *would* find exactly that problem. A number of developers were surprised by how creative testers needed to be.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JR: I am constantly amazed at how easy it is to break systems that provide an interface to the general public. For example, my local grocery store upgraded their software last Friday. On Sunday, when I did my grocery shopping, I whipped out my credit card to pay, and the cashier said, "Oh, if it's a <large-bank-name>, we can't take that today." "Why not?" "The debit part of it doesn't work. You need to use a credit card."

How is this possible? How could the vendor not test (or not realize there's a problem with) the software for a common banking partner? I don't know about the cost to my local grocery store, but you can bet I'll be checking the prices of what I buy. If the vendor can't get the payment details right, how will they get the cost details right? (You and I both suspect that the price lookup is different from the payment verification, but if their vendor got one thing that wrong, why wouldn't the vendor make other mistakes?)

I don't understand how producers can continually *not* test their most common scenarios. These common scenarios aren't just the happy paths; they're the common ways people interact with the system, whether that scenario is a common mistake or a typical job the user wants to accomplish. I'm surprised when testers (and developers) don't break those scenarios down from system level tests to smaller and smaller parts of tests. Knowing that your product works in its most common form gives me a feeling of confidence about the system. And, if I can't prove that the common scenarios work, I have areas I can address in more detail.

DDJ: How would you describe your testing philosophy?
JR: I want to make sure the testers are testing the areas that provide the most value to the customer and the areas where we have the most risk. That means the testers have to work with the requirements people to understand the value to the customer. And, the testers have to work with the developers to understand the areas of risk.

JR: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JR: There are two important things for testers to know: Testers need to know about the systems they are testing and how to be friends with their developers. They need to learn the common product scenarios; the implementation language(s) and how developers can be tripped up by the language; what's most valuable to the customers; how to demand automation hooks from developers; and how to use those hooks to develop small pieces of automation. Testers who ask these kinds of questions also tend to be people who use their previous knowledge of other systems and apply that knowledge to this system.

Testers need to be friends with developers, so they can eat lunch with their colleagues and ask, "What was challenging for you? What should I look at? What should I avoid?" I don't necessarily believe the "what to avoid"--I'll ask why so I can think of ways to test.

Developers need to learn how to test as they go, so they can learn how their code could break. Developers have to learn that they can't just do unit tests; they need to look at larger chunks. There's a huge gap between unit tests and system tests, and most of that gap is the developer's responsibility.

Testers, along with developers, need to continually re-examine their approaches and ask themselves, "Am I learning something new from this testing approach? If not, what do I need to?"

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JR: This whole business of measuring how many defects there are by severity. Oh please. Severity is irrelevant. If you have defects in the code, it's dirty. Seeing one defect means it's ok to have more defects. If testers showed the total remaining open number of defects by week, instead of anything else by week, the trend would have a lot more benefit to the project. Developers need to learn to clean up the code, and the best way to do that is to measure the total number of remaining open defects.

Severity and priority might be useful for organizing when and what to fix, but people start playing the defect promotion/demotion game as soon as they start counting by severity. That game allows people to put blinders on, and release broken software.

Now, it's always a business decision about when and what to release, but the people who make that decision need the data to make a good decision. They can't make good decisions without trend data.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JR: What should project managers know about testing?

Project managers need to know which testing should be done by which people, and if those people are actually performing those tests. A PM who waits for the testers to test the product (of course, in three weeks!) is an incompetent PM. Software is a design problem, not a manufacturing problem. You gotta test the design as you go, which means no matter what lifecycle you use, you have to start testing from the beginning.

Test driven development (TDD) is really about design, and good PMs should encourage their developers to use TDD. But even if the organization is wedded to waterfall, or if they think they need an architectural phase, the testers should know enough to participate in architectural reviews, and design reviews. Testers who bring curiosity and skepticism will help the developers create a much better product.

Sometimes, asking the "dumb" questions help people realize what's going on in the product. I once asked "How does that queue work? I understand how things get into it, but I don't understand how things get out." Turns out, there was a design problem in removing things from the queue. If I hadn't learned enough about systems to know that, or if I had been uninterested in the code, I would never have asked that question.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Adam White 2008-04-08T19:09:09Z 2008-04-08T12:30:00Z tag:,2008:/50.32612 2008-04-08T12:30:00Z Adam White manages Test Engineering and Escalations for PlateSpin. He makes the most of this dual role by using the customer problems he encounters to drive changes in his team&#39;s testing process and thus reduce the need for future escalations.... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Adam White manages Test Engineering and Escalations for PlateSpin. He makes the most of this dual role by using the customer problems he encounters to drive changes in his team's testing process and thus reduce the need for future escalations. As Test Manager part of Adam's job entails engaging his testers in philosophical discussions about testing and living his belief that teaching testing is more about helping people improve their ability to think about software than it is about instructing them in any particular technique or methodology. He also tinkers with neural networks and adaptive systems and such as he experiments with processes for analyzing financial markets and otherwise managing risks of various sorts.

Here is what Adam has to say:

]]> DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AKW: I remember working for a young start-up company near release and we were still finding bugs rapidly. Ship time arrived and the business side decided to ship the software. I was actually told "Stop testing - If you keep testing we will keep finding bugs. We can live with the bugs you have found. We have to ship the software". I realized that in commercial business-to-business software it's about the company's ability to manage the confidence and risk of the customer, not about finding every bug.

DDJ: What is the most interesting bug you have seen?
AKW: Do I have to pick just one? :) A scheduling dialog exhibited some interesting behavior found by Aqiqul Hoda, a tester on my team. The dialog contained a date/time control. If you had the date/time dialog open when the computer time changed to be on the hour an unhandled exception would appear. Aqiqul logged a bug for the unhandled exception without knowing the root cause. All of us thought he was crazy - until he reproduced it later in the afternoon. Eventually we did enough diagnosing and troubleshooting to get to the root cause. It was an amazing bug and great team troubleshooting.

DDJ: How would you describe your testing philosophy?
AKW: My testing philosophy used to be about stopping the shipping of bad software (bad in my opinion). Now my philosophy is focused on managing the confidence and risk of project stakeholders. My primary aim is to provide stakeholders with accurate and timely information so the project team can make informed business decisions about the software.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AKW: One of the most important things for me has been keeping my skills sharp. I'd like people to remember to practice the testing craft not only at work but in everyday life. Let testing permeate your life. Always try to look for new and interesting ways to do things.

I do this by reading a lot and testing random programs that I know nothing about, just for the sake of learning and keeping my skills sharp. I have gotten involved in testing communities and attended dinner meetings and conferences. I also do things that put me outside my comfort zone like improv classes. All of these things link back to my work.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AKW: The biggest challenge I see for testers is finding ways to test smarter. Systems are getting complex with technology, scenarios and interactions. This creates a huge testing space. It's no longer sufficient (and in some contexts never was acceptable) to bang away at the keyboard and hope "bad things happen".

The challenge for the testing discipline is getting fresh, young blood to pursue testing as a career. The testing world tends to get ignored in university curriculums and isn't even considered as a career option by most university graduates.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Joshua Williams 2008-04-06T01:02:12Z 2008-04-01T12:30:00Z tag:,2008:/50.32611 2008-04-01T12:30:00Z mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Joshua Williams  has been a tester throughout his twelve-plus years at Microsoft. That time has seen him working on globalization, testing drivers, and leading the USB support test process, on more architectures and versions of Windows than you probably care to hear about. He has worked with and on various automation frameworks and process improvement exercises as well, which experience is one reason he coauthored the recently published A Practical Guide to Defect Prevention. (See my review elsewhere on DDJ.) His primary focus nowadays is finding ways to help testers better enjoy their work, a mission I wholeheartedly support!

Here is what Joshua has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JW: I had been working in the back room of computer stores for a few years as the "fix-it guy". When I did my first interview loop to start at Microsoft, a test lead asked me to do some tests on an application he then loaded up for me. I am sure I didn't do it all correctly, but I found some problems, and passed the interview. I started by testing hardware compatibility. It was a conceptual leap for me to move from "making the machine work" and "pleasing the customer" to a testing mindset. Obviously pleasing the customer is still important, but in testing we strive to find, understand and log problems, not just find workarounds so we can get the customer up and running. It's the discipline of digging into the problem and getting it fixed that was the change. And it turns out, it's rewarding, too, just in a different way.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JW: After all the technical work, the code, the bug reports, etc., I would have to say that effective communication can make or break your career. I have made plenty of mistakes, and most of them were trying to get the right message across. Communication with your counterpart developers, the program managers, your lead or your employees can create trust, or it can erode it. Sometimes the unintended message is not just a result of what you say, but of how you say it. There are a wide range of strategies people take up when they communication with others. Some people get aggressive, some grow more passive. Your goal is to deliver the message you have in a way that you accomplish your design, not just to say the words. This matters in all kinds of scenarios from defect reports all the way to defending a bug before the release management team. It especially matters when leading a team. Communication can be tough to master, because it isn't a hard skill, but it often carries more weight than your technical know-how.

DDJ: What is the most interesting bug you have seen?
JW: There was this bug that only occurred because of a mix of power management and removable storage. It was interesting to me because it demonstrated how even good test teams can wear blinders. I am confident that the removable storage team did a full battery of tests, probably achieving incredible code coverage results with their testing of removable storage. The same is true of the power management team. But, they never got together. It isn't anyone's fault, but then again, it's both their faults. Should storage drivers and functionality include power management testing? Should the power management test team have a huge array of different kinds of devices and removable media? Where does the responsibility of one team end and the other begin? There aren't perfect answers to these questions. This kind of challenge is part of what makes testing as a career fun: it's not all black and white. To be effective, we have to look outside our boxes, and see the world a little more like the customer does. Only by doing so can we understand how to best test.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JW: Some of the best testing I have ever been a part of was a single SDET and a single SDE focusing on a specific feature in a tightly-coupled feedback loop. They made the most progress because they both shared the same goals: make the feature and the tests great. They both contributed to both goals, even though they owned only one each. This was great because they could communicate openly and honestly. They put away their egos and worked together. The "Quality" wasn't left to the SDET; they both owned it. Not every project will work out like that; but the core principle doesn't change: put away your ego and remember that everyone is responsible for the quality of the product.

DDJ: Is there anything else you would like to say?
JW: People need to read. Testers need to study other disciplines and understand what other companies are doing. I am lucky to be in a group of avid readers, and it has really pushed me to read more. I have really enjoyed reading books on business theory and process management from industries outside software. It has been remarkably educational and eye-opening in how I view quality. I would challenge testers to read. And, they can start with a book that I contributed to: A Practical Guide to Defect Prevention, with more info at www.defectprevention.org. It has a lot of info about software development, but also has a lot of info about things we have learned from other industries.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Selena Delesie 2008-03-11T19:56:42Z 2008-03-11T12:30:00Z tag:,2008:/50.31060 2008-03-11T12:30:00Z Selena Delesie is a photographer and ballroom dancer who enjoys camping around Canada. She's also the Software QA Manager for Intelligent Mechatronic Systems and President of the Kitchener Waterloo Software Quality Association. She accidentally discovered testing back in high school... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Selena Delesie is a photographer and ballroom dancer who enjoys camping around Canada. She's also the Software QA Manager for Intelligent Mechatronic Systems and President of the Kitchener Waterloo Software Quality Association. She accidentally discovered testing back in high school as she debugged her way through programming class. College brought her more testing experience in the form of helping her classmates determine why their programs weren't doing what they were supposed to. After college she continued both the testing and the mentoring for a variety of employers, including a stint at Research In Motion playing withtesting their wireless devices and services.

Here is what Selena has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SD: High school programming class was my first introduction. We were not taught about testing our programs, though of course they had to work. I remember doing iterations of ‘testing’ on my code to get it running correctly. Though I was proud of my creations when completed, I found debugging them to be much more satisfying. Hunting for and identifying problems in software became an enjoyable activity for me the remainder of my computer classes in high school and university. I was often helping classmates debug their software in addition to my own, even through late nights in the computer lab, and really loving it. I imagined myself a software detective, though I didn’t realize I could turn it into a career.

My real introduction occurred when I fell, quite literally, into a software testing job after university. I had applied to everything under the sun to find a job. Somehow my resume ended up in the wrong pile and someone noticed. That someone read my resume and decided to bring me in for an interview on a whim. I got the job, and I was suddenly a software tester. I was excited to discover that I could get paid to do something I enjoyed doing so much. I soon realized that testing could be an art and a craft. The more I tested, the more intrigued I was to find better ways to do it. I was hooked.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SD: It astonishes me that not a single class I had in school taught me how to test the software I was creating. From what I have observed this hasn’t changed very much over the years. Our society relies so much on software, yet hasn’t promoted many of the activities that will help ensure quality software is released. While companies want to release quality products, there are still a lot of misconceptions about what it takes to do so. Software today is extremely complex, so flexibility and a variety of test techniques is key... the 'old school' approaches to software development and testing are not as effective on their own as they once were.

So there is an ongoing need to educate other people about what testing is and is not, and why it is important. Many people (executives, management, development, and even testers) have no idea what testing is about, how much work it is to test software, why it is a valuable activity, and why it is not just an end-of-lifecycle activity. Teaching others about this craft can be frustrating, and it sometimes feels like a lost cause. I continue to do so just to experience those moments when I see the glimmer in someone’s eye that shows me they 'get it'. It’s great to find a network of people who are doing the same all over the world.

DDJ: How would you describe your testing philosophy?
SD: First, systematic yet varied. I have tried a variety of techniques and approaches in my experiences, and I have discovered that they all work in one situation or another. I prefer to plan and organize work for a project as early as I can, and determine which approaches would be most appropriate for it. I have found it important to be flexible as the plan is executed though, as new information and experiences on the project will revise what an appropriate plan of action is. For me it's about understanding when different approaches are most effective, and not being afraid to utilize them.

Second, learn, learn, and learn some more. When I began my career in testing, I knew very little about it. I actively sought opportunities to learn and expand my knowledge on the job and in formal training. While I now make time to read a variety of books, articles, and blogs, my most valuable learning experiences have come on the job and in conversations with people who make me think. I am constantly learning new things and evolving my methods and approaches in testing, which helps keep me engaged and successful in my work. I try to share my love for learning with those I work with in the hopes that they will catch the learning bug too.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SD: Important things for testers to know and do:
  • How to look at something from different perspectives. Using the same perspective (or no perspective) will limit the number of solutions to a problem, as well as the number and type of defects that are discovered. As testers, we need to wear different hats and play different roles. Try pretending to be different stakeholders or users of the software. For example, a tester could pretend to be: a product super-user, an elderly person who is uncomfortable with new technology, the VP of your organization, the IT specialist who will support use of the product, someone who speaks a foreign language, or someone who is deaf. Each stakeholder will care about different aspects of the product. An ability to look at the product from the perspective of different stakeholders will help uncover different problems.

  • There is no 'right' way to test. There are many approaches and recommendations for software testing, but not any one is the 'right' way in every situation. It is valuable to understand and know how to use as many of these as possible, so when it is time to solve a problem or select a test approach for a particular piece of software, a tester can dive into their 'toolkit' and use the most suitable one. A mechanic would not diagnose all car problems with a single approach. Similarly, a tester should not attempt to find all software defects with a single approach.

    Testing with positive and negative functional tests alone is typically insufficient for most software created in this day and age, yet many organizations still believe the contrary. Consider: risk-based testing, orthogonal array theory, persona testing, flow testing, end-to-end system testing, and creating visual test models. These are a small handful of approaches that can be extremely valuable in different situations and will find different types of software defects that standard functional tests would be unlikely to uncover.

  • Learning never ends. Learn as much as possible about anything and everything. If interested in software testing, read about it or take courses. Venture into topics such as critical thinking, problem solving, working with other people, detective work, and even the art of magic tricks. These areas, and so many others, will improve ones' abilities as a tester. I personally aim to learn at least one new thing every day. It keeps me fresh and alert, and helps ensure I don’t get stuck in a rut doing the same things.


Important things for developers to know and do:
  • Unit testing is important! I know many developers who feel the crunch to get software developed and delivered, and in that crunch unit testing gets skipped. The result though is that the software test team ends up spending much of their test cycle finding bugs that could have been found during unit test, and less time finding other types of (often more critical) bugs.

  • Testers are your friend. The last thing a developer wants is to see bad bugs be released in software they wrote. Testers will help uncover some of these bugs on their own, but can do so much more if developers and testers work together. Teach a fellow tester about the software you developed, talk to them about the software design, and even involve them in code reviews. I have worked on projects where developers and testers rarely interacted, and those where they worked closely with one another. The latter experiences were much more satisfying and led to open communication, testers helping find bugs during the design phase (when they are much cheaper to fix), and a more solid product being released.


DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
SD: I still encounter people who emphasize the importance of numbers and percentages, particularly around the number of test cases executed and failure rates. Statistics can be valuable in certain situations, though only if the context behind them is provided. Contextual information is much more valuable to report on what testing has been completed, the testing that hasn’t been completed, specific risks and issues for a particular feature in the software, what failures were encountered, or even the test approaches that have been utilized.

Reporting information like '456 out of 625 tests executed' and '10% failure rate' can give a false impression of the state of the testing and of the software. These don’t tell us anything about what the tests were, what types of bugs they would expose, how important the untested tests are to run, or how critical the failures really are. These numbers are like briefcases...sure you can count them, but without knowing what is inside of them, what does it matter how many there are?

[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Robert Musson 2008-03-06T19:32:41Z 2008-03-04T12:30:00Z tag:,2008:/50.30893 2008-03-04T12:30:00Z Robert Musson has been doing software for twenty-five years now. He spent fifteen of those years at Teradyne working on telecommunications products. He also helped deploy the Team Software Process (TSP) there, which experience he enjoyed so much he later... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Robert Musson has been doing software for twenty-five years now. He spent fifteen of those years at Teradyne working on telecommunications products. He also helped deploy the Team Software Process (TSP) there, which experience he enjoyed so much he later joined the TSP Initiative at the Software Engineering Institute of Carnegie-Mellon University. These days Robert is a development lead on Microsoft's Windows Core Security Test Team, where he manages what he calls the "Department of Statistical Distortions". (Wonder what he means by that? See his chapters in The Practical Guide to Defect Prevention for further details. See my review of said chapters elsewhere on DDJ.)

Here is what Robert has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
RM: My first introduction to formal testing techniques was in a MSSE class at IIT. It was the usual stuff: boundary case analysis, the V model, and so on. But I don’t think it gave me a good appreciation for what testing really means and what it should mean. Testing in software is really all about quality control, that is, finding defects in the software. This is a very dangerous view since it leads to the thought that testing is where quality comes from. The mindset then becomes that testing, and quality, are reactive in nature when in fact testing needs to be proactive. We need methods to predict the quality of the products and to focus effort on assuring the product is ready rather than just pounding away until it seems to be ready.

DDJ: What is the most interesting bug you have seen?
RM: I’ve seen lots of interesting bugs that take down critical systems for extended periods. What makes them interesting to me is the simplicity of the mistake. For example, many years ago someone once put a comment in the code with the "/*" format and forgot the closing "*/". This accidentally commented out an error path. The code worked under all except the most extreme conditions, which of course were impossible to test for. The most insidious bugs are typically simply goofs on the part of the developer. Yet our testing effort tend to focus on ever more complicated techniques thinking that it is the complex race condition or logic error that must be discovered. We often forget about simple things to find simple problems. Code inspections, for example, are often thought of as a development step, but they also should be thought of as a form of test, or rather, quality assurance. In the cited example, the problem was found by someone just looking at the code and not by any complex test that reproduced the problem. Testers should be familiar with the details of development simply so they can discover defects by thinking rather than by brute force. That is really the point of modeling and simulation. Think about the issue and discover problems before they occur.

DDJ: How would you describe your testing philosophy?
RM: Testing is not about finding bugs but about proving readiness. If testers are finding bugs, that indicates a problem with developing. The job of testing is to prove the software should be shipped. Unfortunately, the mindset of finding bugs causes us to ship once our test cases have been run. As a result, software does not so much ship as escape. Most teams have no idea how many undiscovered defects exist in the software nor do they know the potential impact to the customer. As a general rule, teams do not understand the potential costs associated with fixing the undiscovered bugs. This is the software equivalent of product warranty costs. I am really big on tying the test efforts to system costs. A typical software company spends at least as much maintaining the product as they did developing it. Large product teams then begin to collapse under the weight of the rework costs. I call this a software black hole from which no useful work can escape. Only by controlling the risk associated with releasing software will organizations be able to control repair costs.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RM: We tend to focus on advanced testing techniques. Of course new test technologies are important, but I would like to see all testers trained in statistical techniques and measurements. Simple data tracking, like defects per class, provide invaluable information but only when proper analysis is done. We sometimes see people react to data because it seems bad. Only when we do proper analysis can we determine anything meaningful. There are many things about the development process that are counterintuitive and cause us to make wrong decisions. The job of test is to aid the decision making process, and in many cases, only the test team can provide the information. Yet we often ignore the data, or else react only to the last data point and forget long term trend information. I equate proper data management and analysis to good engineering. Testers should use good engineering techniques and we might be able to spread that type of discipline to other aspects of development.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RM: Automation. I say that tongue in check, but humor often has some element of truth in it. There is a current push that everything needs to be automated and more automation is better. Yet when the analysis is done and we look at costs and impacts to the system, automation can be counterproductive. We need the right automation and the right amount of it. Too much is as bad as too little and in some cases worse. This is where simulation and modeling becomes a key to managing costs. We found that we were actually increasing costs by simply automating everything, and what was worse, we actually had lower product throughput without increasing quality. That was one of those counterintuitive things I mentioned earlier. Some areas are better left to good old ad hoc testing. Don’t get me wrong, some areas require automation as the only way to assure quality. But simply writing 2 or 3 times more test code than product code does not improve the quality and will likely end up greatly increasing the costs. This is where proper planning comes in and having your best engineers as testers can really cut product costs and increase quality. There is a view that higher quality means higher costs, and that is just wrong.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Wayne Miller 2008-02-19T21:27:25Z 2008-02-19T12:30:00Z tag:,2008:/50.30504 2008-02-19T12:30:00Z Wayne Miller is a Supporting Architect with the Enterprise Application Architecture Team at ExxonMobil. Which is to say that he spends his time helping people across his company find ways to solve their testing problems. One day he's advising senior... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Wayne Miller is a Supporting Architect with the Enterprise Application Architecture Team at ExxonMobil. Which is to say that he spends his time helping people across his company find ways to solve their testing problems. One day he's advising senior architects on long term testing strategy, the next day he's helping frontline testers with their functional testing, and the day after that he's assisting usability and ergonomics groups innovate in their work. Sounds like fun to me!

Here is what Wayne has to say:

]]> DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
WCM: Johanna Rothman wrote something that changed the way I think about quality, “Shipping your product is a business decision, not a quality decision.” This is how I’ve achieved what Kent Beck calls “Ease at Work.” I think testers can get very frustrated with work because they are passionate about making the product the best it can be. The irony of software is that it’s never done, and you can chase this goal until you burn out. As Johanna suggests, once you focus your testing energy into providing a forecast of deployment risk to the stakeholders – you can find peace in your work.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
WCM: I am always surprised how consistently test effort is underestimated in this industry. Fred Brooks said 30 years ago, “…few allowed one-half of the project schedule for testing, but that most did indeed spend half of the actual schedule for that purpose.” I think that’s still very common today. I struggle with understanding the forces that perpetuate it. I believe that just about everyone understands the value of testing early. I believe that just about everyone expects troublesome bugs to appear as the product progresses. Across the industry, software project managers and their teams have years of experience. Why do we – the industry - keep repeatedly underestimating testing effort?

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
WCM: It’s hard to remember my first introduction. I often feel like I’m constantly being introduced to testing. It amazes me how many niche testing fields there are in software development – it’s not just enough to have confidence that the application works anymore (maybe it never was). Once you’ve really gotten involved with it full-time, you see how much there is that you have to look at to certify an app – security, load, endurance, usability – it’s a long list. Maybe that’s why we underestimate it.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
WCM: Getting developers to be more rigorous about automated unit testing.

I am a big supporter of the concept of Test-Driven Development (TDD). TDD transforms the health of software and opens the gateway to other kinds of tests and QA. I only have anecdotes, but the projects at ExxonMobil that are more disciplined about their unit tests develop faster and with lower bug rates than the projects that do not.

Talking to vendors, I think it is pretty clear that manual testing is still king in the industry, and that has to change. Our systems are becoming more and more interconnected; we have to be able to reliably test these systems as quickly and efficiently as possible.

I think the prize in Service Oriented Architecture demands disciplined test automation to succeed. I don’t believe that manual testing is going to keep pace managing large interconnected systems that are being modified by many people across many locations and groups. It’s not enough that some of our software teams do this well; I think this is a practice that we need to spread consistently throughout our organization.

DDJ: Is there anything else you would like to say?
WCM: One of the new things our testing groups are doing is to start to take a look at the ergonomics of our applications and workflows.

Today, new hires in their early 20s come to work with 15 years or more of wear and tear on their wrists, neck, and back. We want to ensure our employees can maintain the intensity and enjoyment they have for computing when they are 51 or 61 that they had when they were 21.

I’ve appreciated that ExxonMobil takes this commitment seriously and extends beyond just our own internally developed applications. Ergonomics and usability of our applications is a fundamental priority of senior management. We’re working with vendors to develop more usable, ergonomic applications. We continue to extend our network of internal and external ergonomic experts to devise ways to make literally healthier applications and environments for our users.

For us, there is a lot to do and to learn, but developing more ergonomic applications is a win for everyone. Developers take a lot of pride in the effort they put into their applications. They want feedback that gives their applications graceful UIs. Users love transparent applications that allow them to focus on their business problem and forget the tool. Everybody wins from intuitive, integrated applications where no one gets hurt.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Ross Smith 2008-02-21T17:44:33Z 2008-02-12T12:30:00Z tag:,2008:/50.30299 2008-02-12T12:30:00Z Ross Smith has held a variety of jobs in his life: cartooning, guarding jails, managing unions, and that voice on the other end of the line when you call Microsoft Product Support. Then he found testing and hasn't looked back... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Ross Smith has held a variety of jobs in his life: cartooning, guarding jails, managing unions, and that voice on the other end of the line when you call Microsoft Product Support. Then he found testing and hasn't looked back since. As a Tester, Test Lead, Test Manager, Test Architect, and now Director of the Windows Core Security Test Team, he has been involved with Windows and Office for twelve years running. His recently published The Practical Guide to Defect Prevention is one way he aims to share all that experience with the rest of us. (See my review elsewhere on DDJ.)

Here is what Ross has to say:

]]> DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
RS: It’s surprising to me how often people make mistakes. Human error is a fascinating topic - from nuclear disasters and bridge collapses to blue screens and buffer overruns. It’s not just developers writing new bugs, or software specifications that require "testing" per se, but grocery stores with mismatched or missing prices, misprinted ferry schedules, voter miscounts, etc. The skill errors are forgivable - when an infielder misplays a ground ball - but the knowledge errors - throwing to the wrong base - continue to amaze. An example of a knowledge error we tend to make in software development is assuming a new user interface is an improvement - often times it is, but each user has a different "experience" and respecting that diversity is something that needs to be honored in the design phase. To non-engineers, software is not intuitive, and can even be intimidating. What seems like a simple or obvious improvement to a proficient user can be sheer terror to a novice.


DDJ: What is the most interesting bug you have seen?
RS: I don’t know if it was the most interesting, but the most memorable one for me was in my first testing role (COM/OLE for the first version of Windows NT). I ran into a case where a mouse move message was not being redirected correctly if it followed a right button click message. It was tremendously obscure and seemed to happen intermittently, but after a lot of time creating convoluted hardware configurations with the mouse plugged, unplugged, and watching the debugger, the logic error was obvious.

Having spent the last couple years on our book The Practical Guide to Defect Prevention I also have a new appreciation for the simple typo and for the minor off-by-one errors that shift the UI by a pixel and stuff like that - they are only noticeable over time, but can turn out to be huge.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RS: It’s imperative to keep the big picture in mind. Many times, the test team can get so focused on perfection that they miss the obvious things that drive down the users' perception of quality. A large scale test effort might help an API work perfectly and handle error conditions gracefully, and performance is rock solid, but there’s a typo in the intro dialog or an edit box is truncated. Keep the user experience front and center. If a bug exists, but the customer doesn’t run into it, is it really a bug? :) Consider how to prevent the bug from appearing in the first place - invest in defect prevention techniques (such as root cause analysis, failure modeling, customer experience, etc.), partner with development, and monitor user feedback.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RS: It sounds a little odd - but I think testers can get in a box and try to be too thorough. The test matrix continues to explode and it is not usually possible to test everything. Again, step back to look at the big picture, and then assess the risk of missing defects in a certain code path. If a particular block of code hasn’t changed, then maybe it’s OK not to test that for a given release. The use of well known risk analysis, probability, and modeling techniques can help guide the test effort and improve effectiveness.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
RS: "Quality at the source" is a well known phrase that should apply in software development. Manufacturing and assembly line work moved many years ago to empower machine operators or assembly line workers with the ability to improve quality. The parallel for software development is improving the quality of the code as its written. Obviously, things like code reviews and inspections, design patterns, unit tests, etc. are a great start, but again, thinking about the big picture, how do we involve users earlier in the design and coding processes to prevent bugs before they are designed or coded into the software?

DDJ: Is there anything else you would like to say?
RS: Please check out The Practical Guide to Defect Prevention and www.defectprevention.org, and definitely send me any comments, feedback, or suggestions.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Matthew Heusser 2008-02-06T19:22:00Z 2008-02-05T12:30:00Z tag:,2008:/50.30074 2008-02-05T12:30:00Z Matthew Heusser develops software, such as a financial application that handles hundred of millions of dollars of fund transfers each year. Matt tests software too, where "testing" might mean writing unit tests, manually poking at an application, or driving the... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Matthew Heusser develops software, such as a financial application that handles hundred of millions of dollars of fund transfers each year. Matt tests software too, where "testing" might mean writing unit tests, manually poking at an application, or driving the release process for a commercial application which brought in millions of dollars in revenue. And Matt also teaches other people how to develop and test software effectively. You may have read one of his articles here on DDJ Online, or heard him speak at any of a multitude of conferences, or seen him on the SW-IMPROVE discussion list he co-founded.

Matt understands that you may not agree with what he has to say, so he offers only two guarantees when you talk with him: that you'll leave the conversation thinking, and that you won't have been bored in the meantime.

Here is what Matt has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MH: Early in my career, I worked for a company that demoed some software and had an instant sale.

They pushed the demo into production, then brought in the new hire for "maintenance." I was the new hire.

We had no requirements process, no change control process, no version control, no code review, nothing. Oh, and the software didn't work in the first place. All I knew was poke-testing, also known as happy-path testing.

The results, though tragic, were predictable - and painful. I thought to myself "There has got to be a better way" and started learning. Along the way, I found that I actually enjoy software testing; that this task many people think of as dull and boring, when done well, was anything but.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MH: The huge disparity between excellence and mediocrity. A few years ago, Brett Pettichord came up with his concept of "Four Schools of Software Testing", but I believe he missed the biggest one - the oblivious school. So many people in the field today have little knowledge about testing and no interest or incentive to learn. (No, I am not worried about offending them; they don't read blogs.)

I remember when Extreme Programming first became popular, and people like Ron Jeffries and Kent Beck were claiming that testing roles were ineffective and obsolete. I thought to myself "To think that, you must have never met an effective software tester" and that is probably true. They hadn't. The only interaction they had had were with members of the oblivious school.

It amazes me when people talk about test cases run in a day and they count single digits. Of course, that is possible - IF you spend four hours a day in meetings, and IF you create detailed test cases according to the IEEE 829 spec, if you use no automation, and if you use no exploratory or rapid methods.

When I run informal testing challenges at conferences, among my colleagues, if someone can't run ten test cases in three minutes, I'm surprised.

DDJ: What is the most interesting bug you have seen?
MH: The Microsoft Paint Image->Attributes->Height 1, Width -1 bug.

It's interesting because it's an unsigned to signed rollover (didn't those go out with thin ties in the early 1990's?), that it is in a mature product, and that it is probably not worth fixing. It's not worth fixing for a couple of reasons - First, Paint is a giveaway product that demos the features of windows. If you want something solid, get Photoshop or anything else. Second, there is a clear work around; -1 just plain isn't valid; the people that trip it are testers. And third, I hope, it's a great bug for testers when they do demos of exploratory testing.

In that case, I hope Microsoft keeps it around a long time.

DDJ: How would you describe your testing philosophy?
MH: Great Question. My short answer is that when testing is actually running the program under test ("dynamic testing"), you are going through a sort of investigative process of "will this software meet my customer's needs."

Unless the software is extremely simplistic, that turns out to be a rather complicated question to answer. We can never know for sure - there is always one more test to run. So "Are we done answering the question yet?" turns out to be another tough one. Then there's "What do we know with the research we have done so far?"

So I see testing as answering these really tough questions in a creative way. Any tool we can use to accomplish that - scripting languages, driving the browser, FIT, jUnit, Ruby, Windows Explorer, Task Manager, Excel Macros, VBA - any tool we can use - may be helpful.

I don't think you have to be a computer programmer to be an effective tester; I would hold critical thinking as the key skill. I've known many testers who didn't know the testing literature but were wildly effective - in every case, they were extremely strong critical thinkers. Having a CS degree just gives you the ability to do some things *really* *fast* and, in some cases, at all.

I generally split tests into two overlapping groups - developer facing tests and customer facing tests. The Dev tests are TDD and Automated Unit Tests - the Customer-Facing tests are what we now call "acceptance tests", but also "traditional" black box testing. For developer facing tests, I like the idea of a standard regression suite that makes sure we didn't break anything as we change our code. The Pavlovian response to the green bar - "oh, good, I green barred, I can go to sleep now" worries me in some cases however, because in that whole test "cycle", the person didn't do any critical introspective - any critical investigation - all the stuff I defined as testing above.

Also, I don't know if your readers are familiar with the mine field testing analogy, but I'm less and less in favor of running the exact same tests every time. Two common approaches that avoid that are model-based testing and exploratory testing. I kind of run down the middle; I use whatever automation tools that I find valuable in the moment to help me determine the state of the software under test.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MH: I think there are a lot of "red herrrings" out there about software testing. The first is probably the cult of requirements. Look, we've been saying we need better requirements for twenty years; that strikes me as placing testers in the position of the victim. "Get me better requirements" / "I need to be involved up front" / "I need a seat at the table" / "I need management buy in" / whatever. All of those are good things, but essentially, as an industry, we are insisting that other people change, not us. At a recent lunch conversation with Michael Bolton, he said that if he got involved early, that's great, but if not - he still wanted to do the best possible job with the time and resources he was given. I respect that stance.

We need to spend more time talking about how to improve our own skills, and less time asking other people to change to make our jobs easier.

Another one is the cult of documentation, which says that every test case must be documented in excruciating detail, then reviewed. One thing I know: If you spend eight hours a day documentin', one thing you ain't doin' is testing the software.

On the flip side, one thing I think we should *not* ignore, that *is* important, is writing skill (and, of course, testing skill, but I think I already mentioned that). Spending eight hours documenting is bad, but it's worse if you don't have the writing skill to make what you are reading compelling - in that case, nobody's going to read it. Or if you don't have the ability to sift the wheat from the chaff.

It's really easy to document the wrong thing. Then you didn't just waste your time - you've wasted the time of everyone who will ever read your document.

DDJ: Is there anything else you would like to say?
MH: Don't Belong, Never Join, Think For Yourself - Peace. No, wait, that's Victor Stone's Tag Line. Bummer.

I should add, probably to the Philosophy section:

It is so easy to misconstrue and miscommunicate things in the world of software testing. For example, I wrote that we answer questions like "Does the software work?"

Of course, we can't answer a question like that. Instead, on our best days, we can give a rough estimate of the state of the software under test. Likewise, we can't answer "Are we done?" - instead, we can provide information for decision makers, and let them decide whether to take one calculated risk (shipping buggy software) or, possibly, another (missing a market window).

That all depends on our mission - bug hunting vs. assessing quality are two very different missions. However, some missions are better than others. For example, the mission of "verify the software is correct", when the application has anything more than trivial bugs, is a pretty fruitless one. How can you verify a property that is not true?


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Mike Zintel 2008-01-29T20:20:30Z 2008-01-29T12:30:00Z tag:,2008:/50.29820 2008-01-29T12:30:00Z Mike Zintel's life goal is to be a train driving photographer. While he hasn't achieved that yet, he has achieved many other interesting things. Like being told he couldn't plug his fifth grade science project back in after it caught... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Mike Zintel's life goal is to be a train driving photographer. While he hasn't achieved that yet, he has achieved many other interesting things. Like being told he couldn't plug his fifth grade science project back in after it caught fire. Like writing a program to manage underground storage tank inventory. Like helping invent Universal Plug And Play.

These days Mike is Director of Development for a not-yet-unannounced product at Microsoft. Here is what Mike has to say:

]]> DDJ: What is the most interesting bug you have seen?
MZ: I’ll answer a different question. I’ll tell you about the most painful bug. I got a contract to write a billing and workflow system for a law office. The platform was a multi-user MPM/II system. About twice a week, the business data would become corrupt. I held disaster at bay for about two months with a patch up utility. I ultimately lost the contract and the hardware company I was working for failed. Sadly, the same week the customer pulled the plug I found and fixed the interleaved update to a file control block. Even more sadly, I wrote the same style of bug into an embedded data collector that queued data through a local hard drive 2 years later. Concurrency is hard.

DDJ: How would you describe your testing philosophy?
MZ: I’ll assume we are not talking about life critical software. I don’t buy that developers can’t test their own code. I do buy that tests, and developers (and frankly customers) train to correct paths and that diversity will uncover new bugs. I think developers should write correct code. And correct tests. And then run them. With automation. Then an independent engineering organization with a quality mandate should add mostly automated diversity to the mix. The combination of automation, tests and dashboard/reporting should be tuned over time to catch more and more bugs. When the code looks settled and usable in practice, a beta should be run. Not so soon that the feedback is noise due to ongoing code churn and not too late that feedback can’t be acted on. Concurrent with the correctness efforts, automation and tests that validate stability and performance under load should be developed and applied. When the product looks solid, it should be shipped, with telemetry to validate real world experience. Telemetry data should be collected and processed with automation. The entire machinery should be continuously improved.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
MZ: Systems are becoming more distributed and the business cycle is shortening. I don’t think my philosophy above works in this environment. Perhaps some kind of automated “sparse testing” is the answer.


[See my Table Of Contents post for more details about this interview series.]

]]>
CASTing For Participants And Papers 2008-01-31T17:00:21Z 2008-01-25T18:00:00Z tag:,2008:/50.29852 2008-01-25T18:00:00Z If you are looking to attend a testing conference this year, consider CAST 2008. CAST is not your typical conference. While each session does start with you sitting in rows staring at PowerPoints, each session ends with you discussing anything... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog If you are looking to attend a testing conference this year, consider CAST 2008. CAST is not your typical conference. While each session does start with you sitting in rows staring at PowerPoints, each session ends with you discussing anything and everything with the presenter and other attendees, all enabled by a trained facilitator. Sessions are separated by generous buffers of time, so energetic discussions can continue and continue and continue. Imagine a that - a conference where the point is to confer!

]]> If you are looking to present at a testing conference this year, consider CAST 2008. Check out the Call For Papers, then submit your proposal outline by 4 February 2008. That's only a week and a weekend away, so get cracking!

Dunno yet whether I'll be attending. However, Michael Bolton will be. Matthew Heusser told me he is going. Pradeep Soundararajan is flying all the way from India to participate. Jerry Weinberg is keynoting. And that's only the tip of the iceberg!

While CAST will be a lot of adjectives, "boring" isn't one of them!

]]>
Five Questions With Eric Jacobson 2008-01-24T17:28:43Z 2008-01-22T12:30:00Z tag:,2008:/50.29049 2008-01-22T12:30:00Z Eric Jacobson started Life After University as a Teacher Of How To Use Software. That gave him copious opportunities to notice the problems his students had using software and the defects in said software which his students stumbled upon. Eventually... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Eric Jacobson started Life After University as a Teacher Of How To Use Software. That gave him copious opportunities to notice the problems his students had using software and the defects in said software which his students stumbled upon. Eventually he started pointing these problems out to developers and it wasn't long after that until his first official testing gig.

These days Eric works for Turner Broadcasting, where he tests the software that manages which commercials and programming content air when. Hey Eric, schedule me a Star Trek marathon this weekend, will you? <g/>

Here is what Eric has to say:

]]> DDJ: How would you describe your testing philosophy?
EJ: After attending Michael Bolton’s Rapid Software Testing course [RST], I would like to think of myself as a covert RST tester. Covert because not many of my managers dig RST.

My approach is typically as follows but each test session depends on the context.

Select a chunk of AUT [application under test] functionality that devs consider testable, pose questions to business people until I am clear on how they expect the AUT to function, pose questions to devs until I am clear on how they intend the AUT to function. At this point, I usually have some bugs to log before even starting a test. Next, I attack said focused chunk of the AUT. This begins by helping the AUT to do what it feels most comfortable doing. Once it has proven itself to work consistently, I introduce variables (one at a time) that become more complex, until the bugs begin appearing. I let the tests generate from my mind as I witness behavior. I use the popcorn heuristic, and when the bugs begin to stop popping, I end my exploratory session. Finally, I break out my scripted tests, mark them as pass/fail accordingly, and add any valuable ones that were missing. These scripts are fragments, and only contain one step (i.e., do this, expect this). I’m a firm believer in James Bach’s mine field heuristic, and I tend to encourage different paths through the mind field via intentional vagueness.

I spend 20% of my time automating new tests (or fixing broken automated tests). Lately, these tend to focus in areas of performance testing popular paths through the UI and measuring time spans of UI experience and straight service calls. I use a home grown keyword-driven framework wrapped around QuickTest Pro. I’m also fascinated by monkey tests that randomly make decisions on what to do and can run indefinitely, though I only have a few of these and they aren’t good enough to find bugs.

Is that a philosophy?

DDJ: What is the most interesting bug you have seen?
EJ: Unfortunately, I saw this one in production...in an internal HR app I tested. After managers began complaining that performance appraisal comments they had written for their employees began disappearing, I decided to quiz the users and do more testing in said area. I quickly realized the employees could actually delete their manager’s hidden comments by saving new comments of their own. This was a small company and at the time we had no backups to restore. Most managers lost all their free form text for their performance appraisals and didn’t know it until they were about to conduct their face-to-face performance reviews. I was asked to send out an apology email to the entire company for missing that little bug. It was humiliating.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EJ: Though I haven’t read anyone else’s response, I’m sure I’m not alone on this one. Yes, writing manual test cases with details (e.g., UI stuff) is always emphasized and I think it is a major time waster. From my experiences, these test case details are typically ignored upon execution because they are wrong. Which begs the question, should I fix test case details and lose test time or leave them as is, which would leave a trail of incorrect information?

Another typically emphasized metric is the percentage of automated tests. My management loves to set goals like 50% of your test cases should be automated. These metrics are silly because they assume there is value in automating 50% of my tests, which there may or may not be. But I guess if I only write 10 tests, I only have to automate 5 of them. Hmmmm...

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
EJ: You should ask what frustrates me as a tester? I would answer, that there is no stopping point. Developers can sit back and admire a completed app, knowing their code can do what the specs say it should. They only need to provide one solution to a problem. Testers can never sit back and admire a completely tested app because they know if they have more time they can find more bugs. Testers are not providing a solution, we are inventing problems. Yet, as testers, we are constantly answering questions like “Are you done testing?”, “Can we ship?”. These questions are the stuff nightmares are made of.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EJ: As WPF, Silverlight, Flash, and other new UI technologies become more common, there will probably be a period where popular automation tools like QuickTest Pro and SilkTest lag behind in their support. My dev group decided not to use WPF last year, partly because QuickTest Pro did not support it and the folks at Mercury were not aware of any plans to support it.

On the flip side, it appears that Test Driven Development will continue gaining popularity for the next several years. This is good for testers! My dev team currently has over 3500 unit tests, which us testers walk through and use as build acceptance tests. It gives us the piece of mind that a huge chunk of the basic functionality is working. We spend less time chasing the boring environment config problems and basic functionality, and more time looking for the fun, less obvious bugs.


[See my Table Of Contents post for more details about this interview series.]

]]>
Five Questions With Evan Goldring 2008-01-17T17:16:09Z 2008-01-15T12:30:00Z tag:,2008:/50.29048 2008-01-15T12:30:00Z Evan Goldring is a test architect at Microsoft. Today, anyway; before that he was a Group Manager (i.e., he ran an entire product group and managed the people who managed the various disciplines). Before that he was a Dev Manager... mhunter https://i.cmpnet.com/ddj/images/headshots/mhunter.jpg Michael.J.Hunter@microsoft.com Freelancer Blog Evan Goldring is a test architect at Microsoft. Today, anyway; before that he was a Group Manager (i.e., he ran an entire product group and managed the people who managed the various disciplines). Before that he was a Dev Manager (which is to say he managed the people who managed the developers for a product group). Before that he was a tester. And before that he was a developer. And before that he was a tester. And before that...you get the picture, I imagine: Evan has switched between development and testing a lot! He says he keeps coming back to testing for the challenge and creativity. Which I guess means development is rote and boring. <g/>

Here is what Evan has to say:

]]> DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
EG: Like most (all?) developers my first introduction was not because I wanted to test but because my programs had bugs in them. I know I fixed millions of bugs before this, and clearly had to do some amount of testing to find those bugs, but my intro into real formal testing – testing others' code - happened as the result of a bug.

I was visiting with a family friend who was already a professional programmer. I was a Senior in High School and they were helping me wrestle with the question of whether I should major in Computer Science or Computer Engineering. I knew it’d be one of the two, but I really didn’t understand the difference. At some point, he left me to play with some budgeting software he had created for his dad to use. When he came back in the room I told him I found a bug in it. One of the values was coming up negative when it shouldn’t have. He asked if I knew why that was happening and I thought it was pretty obvious and quickly said "Yeah, I’m guessing you are using float instead of a double and I’m overflowing it." What I didn’t realize was that he wasn’t quizzing me, his mind was racing through mathematical and algorithmic errors he could have made but came up empty. It turned out that everywhere else, a float was fine – the values never got that large. But, the total had to be a double.

It was because of that that I landed an internship with his company the next summer where my job was to test desktop software to do mainframe session management. Without being asked, I found automation software to use, devised my own way to measure my "coverage" of the system, and left them with about 80 pages of documentation on the test methodology, instructions on how to run the tests, and a full gap analysis of what I didn’t cover as well as fully documented automation scripts.

They had thought I’d just test the software by using it (ad-hoc testing) and log bugs all summer long. I was under the assumption that clearly this being a professional environment "just" doing ad-hoc testing wouldn’t get me a return job the next summer so I had to step up my game, but I had no idea what that meant so I sort of made it up. Back then, I couldn’t just search the Internet for "software testing". I just thought – if I was in charge what would I have to see from a college intern to be impressed.

I don’t know if that is a statement about how bad they were or how good I was (the company made solid software, made a lot of money, and was bought out by one of the largest software firms for a crazy sum of money before the tech boom so I don’t think they were that bad), but it was a very positive experience for me because I ended up being somewhat self-taught in testing vs. being ramped up by others. By deriving a good test strategy on my own I could better appreciate what others had done in the field by the time I found articles and books on software testing. It’s sort of like learning calculus through memorization vs. derivation.

DDJ: What is the most interesting bug you have seen?
EG: One time I was doing memory leak testing on a streaming media system and found that, under the right conditions we would find an exponentially increasing memory leak happening, but at the same time a similarly decreasing CPU utilization pattern. Since all of us (dev, test, and PM) had inherited this giant mess of code with little documentation, none of us could figure out why we’de see a pattern like this – it just didn’t make sense. Luckily, I had proven myself so the typical route of questioning the tests didn’t happen :).

We set out to figure out what was going on and we found that there was some exponential backoff code that kicked in when the connectivity pattern was in a certain state. That explained the CPU graph, but only further made the memory leak graph unexplainable – you’d expect to see the opposite effect. Well, it turned out that memory wasn’t cleaned up in each case and the algorithm was trying "harder" each time, but waiting longer in between. "Harder" meant doing more work with more connection attempts that weren’t getting cleaned up.

None of this could have been found without collecting a ton of data and graphing them. A picture (created by automated tests), in this case, was worth 1,000 hours of manual testing!

DDJ: How would you describe your testing philosophy?
EG: There are two elements to my philosophy. The first is "Always have a balanced, but wide, test arsenal". If you over invest, you can miss stuff and not be well equipped with the right debugging and diagnostic tools when a certain bug comes around. You have to know when to use automation, know when to use manual testing, and know how to do manual testing efficiently and effectively with "directed ad-hoc" testing. The first thing I do when faced with testing a new component – no matter how big or small – is to create the big picture of what tools I need in my tool box to enable me to attack it from every angle, telling me everything about the system I need. These tools could be anything from software hooks, to well placed “printf’s” (asserts, exception handling, etc), to diagnostic tools, to linear functional regression tests. Once I have the big picture, the second piece kicks in – customer focused testing.

By customer-focused testing, I mean thinking about what sort of bugs each piece, or the depth of each piece, of the picture would find and what functionality it would verify. I prioritize based on the impact that bug would have on the customer. I’ve seen a lot of people, especially junior testers, get excited by finding a corner-case exception but ignore a layout issue as "fit and finish". In reality, the exception could be an extreme corner case – severe, but hardly ever hit – but the layout bug could chop off important text or cause extra mouse clicks in a usage flow that a user goes through 50 times a day. Likewise, looking at how actionable error messages are. Performance (or at least perceived performance) and other characteristics of the software are important to measure. This is how I prioritize the big picture I create and ensure that I build the test tools that will find the most important – from the customers point of view – issues. In other words, I really like to test with the end user in mind.

Once I have the prioritization, I go through and create all the cases as manual ones. That way, at the end of the project, I’ve at least got a record of all the testing I wanted to do and have a backup plan in my back pocket (using an army of manual testers to hit everything). That usually takes on the order of days. Then I start automating, building the tools, etc. Aside from a great backup-plan, this approach has a secondary benefit – you understand the testing you want to do with the automation. The best automation is born from manual processes that were repeated so often that they are well understood and could be replicated easily, without loosing potency, by code. I’m not saying you have to run each test case 100s of times before you can automate, but by at least laying out all the tests as manual ones, it requires you to think through the steps and run through them a few times to make sure they are repeatable and predictable. This small tweak – creating all the tests as manual ones before automating – has saved my teams time and time again either because we can deeply Plan B if some automation doesn’t happen to fit in the schedule or because we thought through the test cases first which saves us from going in the wrong direction and throwing away a lot of work. This is one of the highest return on investment activities you can do on a project.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EG: Testing the parts vs. testing the whole. As I said in my philosophy, you need to find the right balance. I’ve used products, as I’m sure everyone has, that are just dreadful in one way or another. If you talk to individual testers, they will claim their piece is only contributing to that metric in a positive way. However, the sum of the parts is much smaller than the characteristics of the whole. Performance is a great example of this. If you aren’t looking at both the product-wide and the "nuts and bolts" testing of all the pieces that make up the flow of that scenario, you won’t know what the user's experience will be, nor will you know where to look if that scenario’s performance doesn’t meet the bar. I’ve seen many large-scale projects where too much emphasis was put on the parts, but little ownership of the gaps between them existed. This, at least, has been in some of the projects I’ve been exposed to, and this isn’t a comment specific to Microsoft.

This isn’t a direct answer to the question, but I think too much emphasis is often placed on component level testing and I think some of that should be traded for a look at the whole to strike that balance. Using the second part of my philosophy, I’d prioritize against the severity and likelihood of a customer coming across a bug to know what to trade off.

It is admittedly a tricky business. Is a data loss bug that one person out of 10,000 will hit more important to go after than a more benign issue that will cause one out of 1,000 to have to reboot every 10th time they run the program? Is truncated text on a localized build with only 5,000 projected users more important than every users needing 2 more mouse clicks to do a common task more important? At the end of the day, the answer is going to lie in what is motivating and driving the business so the answer could be different in each company, in each product. But, you need to really understand those elements well. Not all bugs are created equal.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EG: We need to learn how to drive quality - in every sense of the word.

Software Engineering is unlike most other Engineering disciplines, in that the notion of "quality" extends far beyond the traditional bounds. Ease of use, doing what customers want and need from their software, easy and unobtrusive upgrades and patches, etc. are all necessary. You find these notions in other engineering disciplines, but few have all of them like software engineering.

I think our challenge is in learning from these other disciplines so it doesn’t take us 100+ years to become as refined as they are in their approach to quality assurance, yet still enable quick turnaround on new features, and driving quality across the entire product development cycle, not as a bolted-on activity that happens near the end. Again, this is not a reflection of Microsoft, but rather what I’ve seen on average across the industry.


[See my Table Of Contents post for more details about this interview series.]

]]>