THE BOOK OF TESTING

Thoughts From a Braidy Tester

August 2008

Michael Corning is a man on a mission. No small mission, either: his goal is to remake testing at Microsoft, and then throughout the rest of the world. Michael joined Microsoft's Engineering Excellence group in part to pursue this mission, which involves patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative testing approaches from across Microsoft and the industry. Dunno whether he will be successful in this endeavor; regardless, I imagine he and his compatriots are in for a fun ride. (Full Disclosure: I am a founding compatriot.)

If this seems an audacious goal, well, Michael is an audacious guy. He is so passionate about politics - both in general and in specifics - that he is Team Lead for the Microsoft Political Action Committee. As a volunteer planning commissioner for his town he is leading a comprehensive for-the-next-century replanning effort which involves patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative technologies from across Microsoft and the industry. As before, while I don't know whether he will be successful in this endeavor, I imagine he and his compatriots are in for a fun ride.

(Can you guess what he talks about in the testing classes he teaches? Yep: patterns applied at every level, concepts ranging from robotics to Aristotle, and innovative technologies and testing approaches from across Microsoft and the industry.)

Here is what Michael has to say:

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
MC: The first thing a tester needs to know about testing is that you cannot test what you do not understand. If follows that anything that enhances a tester's understanding of that which they are testing is paramount. When we refer to testable software, an overlooked attribute is understandability. We've made great strides toward working better with our developer colleagues, to the extent the devs are at least writing unit tests (even more progress is possible with devs who adopt a Test Driven Development approach). Other ways testers can enhance their understanding of any tested artifact is to transcribe the model of the artifact they carry in their head to, ideally, a machine-readable format. If you can't model it, you can't understand it; if you can't understand it, you can't test it. Yet another extremely powerful tool for testers to master are design patterns.

DDJ: How would you describe your testing philosophy?
MC: Funny you should ask. I just submitted a paper to the International Symposium on Software Reliability Engineering entitled, "A Tester's Guide to Aristotle's Theory of Virtue - A Model of a Happy Tester." So I guess my answer to you is the same as the answer I gave to Forbes magazine over a decade ago: I'm an Aristotelian tester. Actually, I'm part Neo-Platonic, too. I am convinced that ideas predate awareness, rather like Plato's "Forms." I call them memes. At the moment, my favorite meme is the pattern language. Actually, I'm beginning to rethink my notion of memes: at first, I saw design patterns as memes; now I'm beginning to think memes are patterns, a subset being design patterns. Put a different way: my work on memetic engineering is refocusing as pattern language engineering.

I am a Certified Public Accountant by training. In fact my Master's thesis in 1980 studied whether financial auditors were auditing around or through the computer. This research found that a majority of auditors considered financial accounting software to be a black box. Only a small minority of auditors used white box auditing/testing techniques. This background continues to inform my view of software testing. In my mind, the tester is an auditor. The function of the auditor is to attest. In financial auditing, the auditor attests with his or her signature that the financial results fairly represent the financial results of business operations. From this attestation, other decision makers make their decisions. Both the auditor and the tester provide information to the eco-system. Investors buy or sell based on audited results; product managers ship or hold product based on tested results.

I'm also a scientific tester. This role is closely related to my audit philosophy. I share the view of Philip Armour that the test org is ideally chartered with finding a class of information that neither dev or pm/ux [user experience] colleagues are tuned to. The tester's job is to identify questions (and subsequently answers) about the application under test that have never been considered. We call them UnkUnks (unknown unknowns). Yes, Secretary Rumsfeld took some heat when he tried to explain the role of UnkUnks in the War in Iraq, but political flak notwithstanding, UnkUnks are, by their very nature, un- or underappreciated. The ironic thing about UnkUnks, though, is that they are pervasive, and everyone encounters them or their consequences, and frequently. So common are the UnkUnks that we have a phrase for them: The Law of Unintended Consequences. Virtually everything else I think about testing comes back to the UnkUnks. Design Patterns play a role in understanding the UnkUnks' potential or presence, modeling squeezes them out in the process of model development, and the philosophical approach I take towards the craft of testing also helps find out things about software that designers and developers didn't or couldn't consider.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MC: Though I wouldn't go so far as to denigrate the following as unimportant, I do believe they are generally misunderstood: test case count and code coverage.

You can see from my previous remarks that I spend precious little time thinking about testing in terms of the number of test cases I have. I do my best to use the machine to generate test cases for me. As George Gilder said, "You make money by wasting what's plentiful." Well, my human bandwidth is far from plentiful. My imagination and intuition, my passion for discovery are limitless. This is why I focus so much of my attention on finding ways for testers to use technology to test technology.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
MC: Two things keep me up at night: where is our catalog of the hard problems in test? How can we leverage technology sufficiently to catch up with the complexity of technology?

The catalog of Hard Problems is our collective record of second order ignorance (viz., we have the questions, not the answers). One of our bigger problems in test is that we're the team in charge of coming out of third order ignorance (UnkUnks), and you can't catalog what you don't know you don't know. You can work on describing the process that you can use to identify unknown questions, but the test community hasn't even done a good job with second order ignorance. As a scientific discipline, we lag behind on two fronts: almost all other disciplines exceed ours in second order ignorance and in the use of modeling.

One of the things that helps me sleep at night when I start to worry about the second thing that keeps me up at night is that many of my colleagues, the best and the brightest, are doing work in test at a deliberately abstract level. Modeling and patterns are the two best examples. As our programming languages continue to help us write test code and infrastructure that take advantage of more and more design patterns, we are making progress. As modeling tools become more ubiquitous, we will make even more progress.

One of the most intriguing things I'm watching is the advent of software that enables robotics. In my view, the Microsoft Robotics Studio promises to bring a whole new level of productivity to testing. But I'm a little ahead of the curve here, so additional comments and observations will have to wait for another time....

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
MC: Legend has it that as Charles Simonyi (Father of Office Word and Intentional Programming) was leaving Microsoft someone asked him what surprised him most about the company. His answer, "How conservative the Microsoft Developer is." I'd say the same thing applies to testers and to test management. For example, it took almost a decade for model-based testing to take deep root in the test community. There are many reasons for this, but being very conservative is one of them. The other thing that constantly surprises me is just how daunting is the task of testing. I heard that someone analyzed all the dependencies in Windows, alone, and the graph used paper forty feet in length. I've seen how far a test tool like Pex can get into the CLR simply by unit testing a resource provider component, and I'm blown away. Put differently, I've been to no other place where it's possible to feel like you're the smartest guy in the room and the dumbest guy in the room - at the same time. Camus was right: the work is absurdly difficult; and therein we find the nobility of it.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Ethan John professes to not enjoy writing code. Testing, on the other hand, he enjoys immensely. That makes him my kind of Computer Science graduate: the kind who codes only because it gives him an excuse to test. Ethan currently works for Isilon Systems, who I am sure is happy to have the advantage of his love for testing.

Here is what Ethan has to say:

DDJ: What was your first introduction to testing?
EJ: I was in school, and got a job as a research assistant on a project called UrbanSim. It was an Agile house, minus pair programming, so they were doing test driven and iterative development in Java. I had only heard about TDD a few months prior, and my initial experiences with it had been positive. Unit tested code tended to work more consistently out of the gate than otherwise, and I was sold after just a few weeks on the project.

DDJ: What did that leave you thinking about the act and/or concept of testing?
EJ: I had never thought about testing as something unto itself before. Previously in school, testing wasn't something you did consciously so much as it was the necessary evil of debugging something that should otherwise work. Getting down and dirty with JUnit allowed me to see the value in testing as a practice, distinct from just writing code.

It also set me on the path toward testing as a profession. I enjoyed the process of writing unit tests far more than I enjoyed the process of writing the actual code. Thinking through the various kinds of test cases and writing them always left me bored with the process of writing code that actually did the heavy lifting.

It actually took me a while to actually learn that testing was a field unto itself. That seems so obvious now, but the University of Washington teaches a very theoretical CS program, so the various fields available to us as CS majors weren't exactly written down. Even though that first experience with Agile turned me onto testing, I had no idea that it was something I could do for a living.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
EJ: Well, I started out as a developer (or rather, as an aspiring developer), and a lot of the testers that I've worked with don't appreciate the value of system design knowledge. Knowing how a system works as a whole, and having deep technical knowledge about how the code works, is extremely useful for everyone from folks writing white box system acceptance tests to those running through black box Web UI scenarios.

Having a large-system view is important for a lot of reasons. It's useful when debugging ancillary problems that might come up. It's useful for evaluating overall product usability, and for recognizing oddities that developers might dismiss as necessary evils. It's also useful for problems that might arise in ancillary areas. At Isilon, our systems are extremely complex, and early in testing, it's common for item X to malfunction while you're looking at item A, and having a general view of the overall system helps tremendously when investigating those issues.

Some of this system familiarity simply comes with exposure to a product, but there is no substitute for knowing what's going on at a slightly deeper level. Knowing what pieces of the system use which libraries, for example, can be invaluable when trying to figure out an elusive repro, or when trying to write a test plan for a fundamental system design change. Testers in general need to be interested and motivated to learn as much as they can, in as much detail as they can, about the systems that they are working on.

This speaks to something that developers need to be good at -- explaining changes that they make and code that they write to testers. I know it's cliche at this point, but testers and development need to have a very close relationship, and part of that relationship is having a common understanding of the scope of code changes and how they might affect the system as a whole.

DDJ: How would you describe your testing philosophy?
EJ: I view testing as science. Hypothesis, test, conclusion, and repeat. This implies a general view of good testers and how they work. Biologists can't simply pawn off their experiments to people completely inexperienced in the ways of the lab -- there is a baseline of understanding necessary to do anything in a biology lab. Likewise, there is a baseline of knowledge and a scientific mindset necessary to test.

Additionally, testing should be approached as a science. Test cases should be framed in terms of hypothesis/test/conclusion. Sometimes these things are implicit -- if you're testing a Web UI, many of the hypotheses are so common that they needn't be stated. But in the world of system acceptance and other more complicated areas, these hypotheses are often not so well defined, and without them, you often end up testing something that answers no question and provides no data.

Finally, science is constantly informed by history, and testing is no different. Historical results are key to determining how to test in the future. If it provided no benefit to test certain things in the previous cycle, how much benefit is it really likely to provide this time around? Historical data informs future results in the test lab just like in the labs of all other hard sciences.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
EJ: Testing-as-science seems obvious to me. The common test methodologies are rigorous and extremely scientific. I'm surprised by the number of testers that don't see it this way. Many people seem to view testing as something rote (perform these steps, record this result, move on) or as something random ("cowboy" testing, extreme reliance on ad-hoc testing, fuzzing as the whole solution, etc.) rather than as an iterative process that generates a wider view of the stability and usability of a system.

DDJ: What is the most interesting bug you have seen?
EJ: Bug 5112 in our system here at Isilon is about the fans in our nodes being prone to failure upon introduction of "foreign matter." A tester with rather long hair had been standing in such a way that his hair was pulled into the node, causing the rear fans to stop working, and the node to overheat. It was closed WORKSFORME by a developer with no hair at all.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Amol Kher has been a tester at Microsoft for five years and change. While his feature teams appreciate his contributions to their work, testers across Microsoft appreciate his contributions to theirs: as one of the authors of a popular internal model-based testing tool, Amol is directly responsible for powering up their testing efforts. Amol has been powering up testers in a different way of late with his recent move into people leading, which he is discovering to be every bit as challenging as writing his model-based testing tool has ever been. He has a new challenge at home as well in the person of his infant son.

Here is what Amol has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
AK: When I worked as a developer in a startup with very little test resources. I can't say I was a well-behaved developer, but I thought about the plight of a web application tester and the difficulties in robustness they had using third-party Web UI tools to automate testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AK: It's never enough, also that people seem to think that because there is very little code, that there are not many bugs. Well what about missing code? Also how close testing is to representing your customers and being that first line of defense.

DDJ: What is the most interesting bug you have seen?
AK: I can't say I have rated any particular bug as the most interesting. Bugs that generally triggered QFE's have been quite interesting in general for me but more from the point of view about why did we/I miss it, what did the customer do that I didn't think about.

DDJ: How would you describe your testing philosophy?
AK: Think like the customer does and try out crazy things with your product. Use modeling to better understand and explore interactions between different components under test.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AK: Time is always short, developers always want more features, customers don't always know how to use the features; how do you balance all of that by providing your feedback? Developers should do component level testing and leave the integration and scenario testing to test teams. Our developers should not just write code, they are responsible for writing high quality code, and they are the best people who could avoid bugs by writing unit tests.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
AK: We depend too much on the number of bugs found and lines of code covered as the only metric of quality. I am generally more interested in data coverage and did you verify the method did what it was supposed to do.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AK: Effectively representing your customer when software code is becoming so complex.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
AK: Model-based testing- Why have I seen the light? I just recently found 2 bugs in product code that has been shipped for over 10 years by not even specifically looking for these bugs. Models can do that to you. Models are what developers will hate to love or love to hate.

DDJ: Is there anything else you would like to say?
AK: Enjoy reading your blog!

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Scott Barber has done a lot of stuff for a lot of people, especially people who test for a living. Especially for people who test performance for a living. Books, directorates, co-founderships, trainings, and the-application-doesn't-work testing - Scott has done it all. Back in the day of Windows NT 4.0, he took the time and trouble and tests to become a Microsoft Certified Systems Engineer. These days, Scott has effectively self-certified himself as a World Renowned Performance Testing Expert via his plethora of books and articles and classes on the topic, all of which arose from his general passion for making performance testing more understandable and more fun for testers everywhere. Maybe even you!

Here is what Scott has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SB: One Sunday evening I got a call from my manager...

"Scott, be at the Marriott Hotel Conference Room at 8am tomorrow. Our CEO has an announcement to make."

Being a good employee, I did as I was told and went to the meeting. As it turns out, our software development company was "merging" with a huge media company. We were going to become their "development branch". As a result, we were giving up our bonuses, overtime, and our base pay was being reduced, but we were going to get stock options.

As one might imagine, I was less than enthusiastic about this. I called my friend of 10 years before I even left the parking lot of the hotel. I told him the circumstances and asked him if he'd help me update my resume, to which he responded:

"Dude, that sucks. Send me your resume I'll take a look, but, we need performance engineers. You'd be a perfect fit."

I said "Performance engineer? What's that?!?"

He replied "Don't worry, you'll like it."

He was right. I've been a tester ever since, and continue to love it.

During my first few projects, I learned just how difficult it is to develop really good software, how challenging it is to find and focus on defects that matter, and how much I enjoyed trying to solve the technical, political, social, schedule, resource, and business problems embedded in all testing efforts. I didn't know it at the time, but I was quite lucky, because as a performance testing consultant I was almost always treated as a peer to the lead architect. In retrospect I realize that was because I was a generalist who understood how systems, not just components, worked from the bottom up as well as from the UI down, who viewed my role as a service provider to the development effort, not as an approval gate on the path toward release.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SB: Over the years I've been shocked to find out that most testers and development groups view testing as a type of QA/QC function. I was shocked to find out that there were testers (as it turns out, lots of them) that were expected to assess "Pass/Fail", to sign-off that a piece of software was "ready to ship", and to generally make business decisions about software and applications, often without even having access to the business reasons for developing the software in the first place. To me, the notion that a tester/test manager should be gating the ship/no-ship decision is baffling. We (testers) are information providers, we are not product managers, account managers, or Vice Presidents of Software Development.

Asking testers to make these "ship/no-ship" decisions, feels to me like asking the CSI's investigating a case to also serve as the judge and jury for that case without so much as a hearing. It seems to me that testers should be providing their results and analysis to the team, the stakeholders and the decision makers so that they have the data they need to make informed technical and business decisions.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SB: I think that the single, most important, thing for a tester to know is where they fit in the team, the project, the company, and what they can do to best serve their stakeholders. I think that the vast majority of testing is done for 1 or more of 4 reasons, but that most testers are not made aware of what their test results are primarily being used for. I think that understanding which one(s) of the following are the primary uses for their test results can dramatically improve the effectiveness of a tester or test team:

1. To provide information to stakeholders and decision makers so they can make informed business decisions.
2. To provide information to developers that help them build better applications.
3. To figure out what problems/challenges/concerns end users will have with the software and communicate that back to the team.
4. To try to assess compliance of the software or application with legally binding contracts and/or regulations.

I'd like to tell all developers that testers, at least testers in my community, are there to help them make better software.

I'd like to tell all developers that most of the time, these testers are no more interested in defects getting reported to management than they are. That defect reporting is a process "thing", not a tester "thing". That I've met very few testers who aren't happier to skip the bug reporting altogether and just sit with developers and help them "get the bugs out" before the bugs ever get on management's radar.

DDJ: How would you describe your testing philosophy?
SB: As a tester, I am a service provider. The service I provide is quality-related information. The people I serve are developers, stakeholders, end-users, and compliance & regulatory agencies (in that order when given my preference). If I do a good job helping developers build quality software that meets business, end-user, and compliance needs, I've done my job well. When I don't get to work directly with developers, if I can identify issues that threaten business goals of the software early enough to do something about them, I've done my job well. If an application goes into production and the end-users don't experience any issues that I haven't previously identified, I've done my job well. When legal regulations and/or contracts are involved, if I can identify areas that are out of "specification" early enough to get them in "specification", I've done my job well.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
SB: I think there are a huge number of challenges facing both individual testers, and the test discipline over the next 5 (and probably more) years. That said, I think that most of those challenges fall into three broad areas.

• Attracting (and Retaining) Good Testers – I recently had the opportunity to meet and work with some software testers in several Latin American countries. Every one of the testers I met are well educated, well trained, well respected members of the team who went to school to become testers, and are proud to be testers. This is simply not the norm. Outside of Latin America, most testers I meet are poorly trained, under respected, of mixed educational background and frequently frustrated with their career. This is a problem. How can we expect to attract folks into testing if so many of our current testers are discontent? I believe that we, testers, need to start viewing ourselves differently if we hope to attract and retain high quality testers. I believe that if we start viewing ourselves as, and acting like, information providers and trusted advisors to software projects, that we will earn greater respect, that our pride in being testers will increase, and that ultimately we will attract and retain increasing numbers of high quality testers.
• Educating Testers – Most testers receive all of their training on the job. When they do get external training, it's generally delivered in short 1-3 day courses with no tests, and no homework, that are well known to be a reasonable way to expose someone to new ideas, but lousy at actually teaching anything. The truth is that testing is a wildly diverse discipline that requires the application of knowledge from a wide variety of fields and that most of the best software testers have taught themselves much of what they know that makes them very good testers. We, testers, need to pay more attention to what skills are common across very good testers and then figure out how to teach those skills.
• Evaluating Testers – I believe we need to fundamentally change how we evaluate the quality and value of testers, especially during the hiring process. Recruiters, hiring managers, and testers widely agree that today's assessments based on certifications, vendor delivered training, question-based interviews, and laundry lists of technical skills are fundamentally lousy indicators of whether or not someone is a good tester and whether or not they will be a good fit within a particular company or team. We, testers, need to get involved in devising better methods. Mismatches between the hire and the role hurt everyone, and currently, the most effective method we have to get a good fit is the old adage "It takes one to know one."

The Association for Software Testing (AST) is sponsoring projects and programs related to each of these areas, but these kinds of changes take time and lots of work. If you are serious about software testing, I encourage you to join AST and volunteer to assist with the projects or programs that you believe will most help testers individually and collectively face the challenges the future holds for our discipline.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |