THE BOOK OF TESTING

Thoughts From a Braidy Tester

May 2007

Rex Black consults on software, hardware, and systems testing around the world, helping major companies do everything from train their testers to offshore their testing. Organizations like Dell, and Bank One, and the US Department of Defense. His clients seem to think Rex knows what he's talking about, as do the hordes of people who attend his keynotes and presentations at multitudes of international conferences and workshops. As do the thirty-five thousand people who have purchased his books Managing The Testing Process, Critical Testing Processes, Foundations Of Software Testing, and Pragmatic Software Testing.

Rex is also President of the International Software Testing Qualifications Board and of the American Software Testing Qualifications Board. Which will either increase or decrease your opinion of him, depending whether you are for or against certifying testers. <g/> Here is what Rex has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
RB: I first got paid to work as a tester when an independent test lab hired me to create automated Unix test scripts and programs in 1987, but I had been doing automated unit testing of my own code as a programmer for years. At first, my thinking about testing was mostly tactical and superficial. It took me a while to realize the complexity of the problem that testing is dealing with. It was about the mid-1990s, when I was developing concepts that would eventually become risk analysis and risk-based testing as described in my book *Managing the Testing Process*, that my understanding of testing both technically and from a business perspective matured.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
RB: How many people expect testing to be a shield rather than a filter.

DDJ: How would you describe your testing philosophy?
RB: I don't know about philosophy, but my preferred test strategy is an analytical risk-based one. Analytical in that it involves up-front analysis, during the early stages of the project, when it's still possible to prevent bugs. Risk-based in that we identify risk items, assess the level of risk, and then allocate testing priority and effort based on the level of risk. In addition, I'll mention that, to me, risk-based testing addresses both business and technical considerations.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
RB: I don't like to think in dualistic, black and white terms like is/is not important; I prefer to think about gradations. I would say that, like the rest of software engineering, we tend to rely too much on silver bullets, and those are over-emphasized. An example would be GUI-based test automation to the exclusion of other good automation techniques.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
RB: For individual testers, it will be outsourcing and globalization. The commodity tester--one who can't distinguish their skills from those of others--will be driven out of the field by cheaper people in industrializing nations. For the testing discipline itself, I see bringing common testing practices in line with best practices as the biggest challenge. Best testing practices lead common testing practices by about 30 years, which is a gap far worse than what we see in programming.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have verified you have checked all of the following. Yes, this is a bunch of work! Make this list your own: remove items which are not relevant in your context - and add items which are unique to your context! Triage items off the list as unimportant, or prioritize them lower in priority. Most important is that you consciously think about and decide what to do with each item. (And if you're a developer, check out my (much shorter) You Are Not Done Yet: Developer Edition checklist.)

• Input Methods
• Files
• Filenames
• Filename Invalid Characters and Error Cases
• File Operations
• Network Connectivity
• Alerts
• Accessibility
• Text Accessibility
• Menus and Command Bars
• Dialog Box Behavior
• Dialog Box Interactivity
• Dialog Box Look And Feel
• Text Entry Fields
• Undo and Redo
• Printing
• Special Modes And States
• International Sufficiency
• Localization
• APIs
• Platform
• CPU Configurations
• Hardware Configurations
• Security
• Dates and Y2K
• Performance and Stress
• Application Configuration and Interoperability
• Setup
• Setup Special Cases
• Upgrades
• Documentation

Posted by The Braidy Tester at 07:30 AM  Permalink |

Although Sara Ford is currently the Program Manager for the Visual Studio Power Toys, she started life at Microsoft as a tester. Back then she drove accessibility into Microsoft Visual Studio' s shell (i.e., all that UI you see as you use Visual Studio), most parts of which she owned testing for at one point or another. Nowadays she spends her time convincing teams across Microsoft to share code with y'all via shared and open source.

Sara spends her free time doing sports of all kinds, including riding her custom-built bicycle. Sara says her goal in life is to be a ninety-seven-year-old weightlifter, so that she is featured on the local news. Here is what Sara has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SF: My first real introduction to software testing was actually during my interview at Microsoft. During the Exchange interview in the morning, they asked me to test a simple weighted network (think traveling salesman style question). The interviewer explained to me that my answer was actually trying to implement the network, and not test it. She gave me a few examples of what she was looking for, and I thought, “whoa, I never thought that way before.” It was similar to only having a pencil, then getting a box of 8 colored crayons to work with. I started to get the feeling to this “software testing” thing throughout the rest of the loop, and by the time I got to Visual Studio that afternoon, I was honestly having fun answering the questions. Originally, SDET was the last thing I listed on my list of positions I wanted, but I left the interview loop jazzed about being a SDET. Equally important, the last interviewer said something that has stuck with me ever since about software testing. He said, “software testing is all about doing the optimal amount of testing in the minimum amount of work.” So this “fun” problem of “how many ways can you break something” became a challenge of “how many ways can you break something as fast as possible.”

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SF: I was most surprised the day I realized the paradox of, “so, um, how am I going to write tests for the tests that I’m writing?”

DDJ: What is the most interesting bug you have seen?
SF: I love this question =) One of the coolest bugs (yes, a die-hard tester will call the bugs they’ve found “cool”) I’ve ever found was in Visual Studio 2005 Tool Window Docking (sorry Chris, my former dev, for sharing this story). In a pre-beta release, if you re-docked a tool window to the same place over and over again, about maybe 100+ times, the window size would start to shrink (as if the user resized the window) until the size actually became negative, eating into the title bar. And no, I didn’t find this manually. I actually stumbled upon this while experimenting with model-based testing, when one of my models got away from me.

My other all-time favorite bug had to do with a screen reader. If Visual Studio had more than 500,000 characters opened in a file, and a screen reader was running, Visual Studio would crash. Took me 10 days working with a customer who was blind to figure out the root cause of his crash. But persistence prevailed in the end.

DDJ: How would you describe your testing philosophy?
SF: Every tester has their own techniques and styles. But, if I had an actual philosophy, I would have to say it is letting Murphy’s Law work for me. I seem to have really dumb luck at times with software. Maybe things don’t come as intuitively to me, or maybe I just see UI slightly differently from the rest of the population. But whatever it is, the bugs always seemed to find me, instead of the other way around. Software testing was such a great job, until Murphy’s Law found a way to one-up me. The bugs would find me only once, such that I could never get a repro (i.e. getting the reproduction steps for the developer to investigate the bug). Sigh, Murphy’s Law.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
SF: From time to time, I hear testing philosophies of “100% automated testing – no manual tests.” I couldn’t disagree more. First, automated testing can never truly replicate the end-user experience. It’s critical for a tester to know their customers, to use the features as they would, and to share their pain. Automation can never catch end-user pain points, even at 100% code coverage. Secondly, manual testing is where a tester’s creativity comes into play. As I said above, the bugs find me, since I can never seem to use UI the way it was designed.

Posted by The Braidy Tester at 07:30 AM  Permalink |

My You Are Not Done Yet list tends to used by testers more than developers, whose eyes seem to glaze over as they start to comprehend its size and detail. One of my missions in life is helping developers learn how to test their code. I have worked with a lot of developers over the years, and I've found that they generally do want to test their code and simply don't know how to go about doing so effectively.

Testers use my You Are Not Done Yet checklist as a final check before they decide that they are done (enough) with their testing. Similarly, this You Are Not Done Yet: Developer Edition checklist is for developers to use as a final check before they declare their code done (enough) to hand over to their testers.

• Customize This List: If you get a bug, determine what test (or even better, what general class of tests or testing technique) would have caught the bug, then add it to this list.
• Use Your Tester: Brainstorm tests with your tester. Review your (planned/actual) tests with your feature team. Coordinate your testing with your tester, especially with respect to tests they have already written/are currently writing.
• Focus On Your Customer: Think, "Where would the presence of bugs hurt our customers the most?", then let your answers drive your testing.
• Test Around Your Change. Consider what it might affect beyond its immediate intended target. Think about related functionality that might have similar issues. If fixing these surrounding problems is not relevant to your change, log bugs for them. To quote a smart person I know, "Don't just scratch it. What's the bug bite telling you?"
• Use Code Coverage. Code coverage can tell you what functionality has not yet been tested. Don't however just write a test case to hit the code. Instead, let it help you determine what classes of testing and test cases that uncovered code indicates you are missing.
• Consider Testability. Hopefully you have considered testability throughout your design and implementation process. If not, think about what someone else will have to do to test your code. What can you do/do you need to do in order to allow proper, authorized verification? (Hint: Test Driven Design is a great way to achieve testable code right from the get-go!)

• Ways To Find Common Bugs:
  • Reset to default values after testing other values (e.g., pairwise tests, boundary condition tests)
  • Look for hard coded data (e.g., "c:\temp" rather than using system APIs to retrieve the temporary folder), run the application from unusual locations, open documents from and save to unusual locations
  • Run under different locales and language packs
  • Run under different accessibility schemes (e.g., large fonts, high contrast)
  • Save/Close/Reopen after any edit
  • Undo, Redo after any edit
• Test Boundary Conditions: Determine the boundary conditions and equivalency classes, and then test just below, at, in the middle of, and just above each condition. If multiple data types can be used, repeat this for each option (even if your change is to handle a specific type). For numbers, common boundaries include:
  • smallest valid value
  • at, just below, and just above the smallest possible value
  • -1
  • 0
  • 1
  • some
  • many
  • at, just below, and just above the largest possible value
  • largest valid value
  • invalid values
  • different-but-similar datatypes (e.g., unsigned values where signed values are expected)
  • for objects, remember to test with null and invalid instances
• Other Helpful Techniques:
  • Do a variety of smallish pairwise tests to mix-and-match parameters, boundary conditions, etc. One axis that often brings results is testing both before and after resetting to default values.
  • Repeat the same action over and over and over, both doing exactly the same thing and changing things up.
  • Verify that every last bit of functionality you have implemented is discussed in the spec and matches what the spec describes should happen. Then look past the spec and think about what is not happening and should.
• "But a user would never do that!": To quote Jerry Weinberg, When a developer says, "a user would never do that," I say, "Okay, then it won't be a problem to any user if you write a little code to catch that circumstance and stop some user from doing it by accident, giving a clear message of what happened and why it can't be done." If it doesn't make sense to do it, no user will ever complain about being stopped.

Thanks to all who contributed to this checklist, especially the SHAPE forum, where this checklist spawned a thread jam-packed with great advice for both developers and testers.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Shrini Kulkarni is a passionate tester who happens to be from India. When he's not actively testing software he's getting other people excited about testing software; catch him presenting at a software testing conferences or peruse his blog Thinking Tester for a taste. While some people choose to go deep in a specific industry or role, Shrini has instead opted to go broad, and so he has worked in a multitude of industries including banking and commercial shrinkwrap software, in roles ranging from software developer to configuration manager to his current job as Senior Test Manager at iGATE Global Solutions in Bangalore.

I enjoy talking with Shrini because his background is rather different from mine, and so I find his point of view makes my own biases and automatic assumptions more visible. His colleagues call him a sticky arguer. He calls himself life time tester. Here is what Shrini has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
SK: My first introduction to testing was to test a supply chain management product where I was hired as a developer around 1998. The company thought that best way to train a new hire in product knowledge is to put him or her in Testing. After some initial hiccups, I could find a few interesting bugs in the product in the areas that were not modified for that release. After 2-3 months of testing, I was assigned to a development project.

That brief stint with testing introduced lots of new things to me about testing. At first, testing appeared to me, like a “proof reading” – double checking someone else’s work. Like others in the industry, I aspired to become a software developer but was constantly pulled for doing “quick” testing due to my raw but effective testing skills - ability to notice subtle bugs in a quick time. Finding bugs in software appeared to me like fun – somewhat similar to a toddler playing with his new toy. Then came a time when I decided to take “testing” as my full time career – something that I enjoyed doing. That was around 1999 – since then I did not look back.

During my last few months of working at Microsoft 2003-2004, I started reading the works James Bach, Cem Kaner and Michael Bolton. I would say that is a turning point in my career in Testing. I learnt that I need to be a better thinker if I want to be a better tester. Today whatever success I have achieved by being a software Test professional, much of I owe to these great thinkers in testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
SK: Ignorance about testing and hence the lack of focus on “better thinking”.

I have seen people approaching testing as rote activity of comparison – expected results to actual results. This is trivializing testing. Testing is treated as a simplified act of checking or a set of repeatable actions. In reality, testing is a much more complex and highly intellectual activity. It is because there are more examples of “bad” testing to be seen than “good” ones. There is a need for education and making people aware of testing and its benefits when performed “correctly”. Even now, I am pained to see people considering testing as a “necessary formality” to be completed before software ships.

DDJ: How would you describe your testing philosophy?
SK: At its core, Testing (questioning, search, investigation) is in itself a philosophical thing. I see a solid line connecting testing and philosophy – Search for an idealistic entity. Like a yogi or enlightened soul, a tester while performing testing goes in search of problems, anomalies in the same way as philosophy goes for search of truth. As Cem Kaner puts it, modern software testing is evolving into a discipline of social science. To be good at Testing would mean having an eye for everything that affects human- computer interaction - Science (computer science or otherwise), Engineering, Economics, Business, Human psychology and others.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
SK: According to me, the most important thing to know about testing is that “Testing is a mental discipline – it is about thinking, questioning – it's all in the mind”. If you ask a typical tester what he/she needs to know about testing – you get answers like “knowing programming”, “knowing about business domain”, and “knowing software engineering processes, Quality models etc”. It is rare to see someone saying “I need to think better” to do good testing. That is the point. Good testing emerges from Good thinking. As testers we must learn and invest time in improving our thinking, questioning and problem modeling and solving abilities – the rest all will automatically fall in its place. Reading about Lateral Thinking, General Systems Thinking, Epistemology etc are few good ways to sharpen one’s thinking skills.

Tester to do: Invest time in enhancing your thinking, investigating skills.

Developers need to consider their tester as their friend – some one who can make them look better. Developers and testers often have conflicting but (ironically) complementing skills. An effective collaboration between developers and testers can help in producing “better software”.

Developer to do: “Support and collaborate” with testers – they supplement you.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
SK: I would not call anything “unimportant”; a skilled tester can always mine “good stuff” from any source of information. However, there are a few things which are outrageously “overpriced” in the current market of Testing. Process, domain knowledge and ability to code are being emphasized as without them there can not be any good testing possible. I often hear people saying things like “This is a technical position, we would want all our Testers to be technically sound", and I agree that having knowledge of these can help a tester but I am against “glorifying these”.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
SK: Scarcity of human resources, skilled testers – challenge will be of finding real thinking testers among “testing robots”.

Increasing awareness about testing is the biggest challenge. Certain sections of testing community have constantly projected testing as “routine” “factory – assembly line” kind of activity. Testers today are not putting efforts to learn about testing and organizations are forcing testers to become “robots” or “button pushers”. It is serving their purpose now but it can not support them for much longer. When they need testers who can think, they will not find any as the market will be full of “testing robots”. At this time, few privileged ones, the “thinking testers” will be really in demand. Skilled testers will rule “software space” - is my prediction.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have reviewed all documentation, a) to ensure that it is correct, and b) to help generate test cases. I have lost track of the number of help sample code listings which did not compile due to one problem or another. I have seen documentation which depicted UI which was not in the actual application, and encountered UI in the application which was nowhere to be found in the documentation. Other collateral can be useful to review as well - all those product support calls for the previous version of your application, for example. And have you looked at your source code lately? Source code reviews are a simple way to find those places where supposed-to-be-temporary message boxes and other functionality is about to be shipped to paying customers. And on and on and on.

• Review postponed and otherwise not-fixed bugs from previous releases
• Review product support issues from previous releases
• Review error reports submitted by customers for previous releases
• Verify each error message which can be presented to your customer is accurate, easily understood, and understandable
• Verify each input validation error message refers to the correct problem
• Verify all tutorials are correct: the steps are correct, UI matches the actual UI, and so on
• Review every help topic for technical accuracy
• Verify each piece of context sensitive help is correct
• Verify every code sample functions correctly
• Verify every code sample follows appropriate coding standards and guidelines
• Review all source code for
  • Security issues (see the Security YANDY list for more details)
  • Potential memory leaks
  • Dead code
  • Correct error handling
  • Use of obsolete and banned function calls
  • Compliance with appropriate coding standards and guidelines
  • Inappropriate user-facing messages
• Verify you have discussed the feature design and target users with your feature team
• Verify you have asked your developer which areas they think could use special attention
• Verify you have discussed your feature with your product support team
• Verify you have brainstormed and reviewed your test cases with your feature team and with your Test team
• Verify you have discussed cross-feature implications with your feature team and with your Test team
• Verify you have completed all appropriate feature-specific testing
• Verify you have completed all appropriate cross-feature integration testing
• Verify you have completed all appropriate real-world use-it-the-way-your-user-will testing

Posted by The Braidy Tester at 07:30 AM  Permalink |

Fifteen years at Microsoft hasn't softened James Rodrigues' Massachusetts accent any. Back then he was a Software Development Engineer in Test, which in Microsoft parlance means "a tester who could be a developer if they wished". Eventually he lost his way and moved into management. <g/> He's redeeming himself, however, in his current position as Microsoft's Director of Test Excellence. A high-level executive charged with making testing better - that seems pretty cool to me!

Here is what James has to say:

DDJ: How would you describe your testing philosophy?
JR: I have a “whatever it takes” sort of mentality. I encourage everyone to feel responsible for all aspects of the product and find ways to fill the gaps rather than complain about the other discipline. Have your complaints lead to actions you can take to improve the situation. It’s an action/results oriented philosophy.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JR: I’d like everyone to answer these two questions as part of their requirements and design:
    - What’s the compelling customer scenario this enables?
    - How are we going to test it?

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JR: Too often I see an emphasis on the quantity of bugs a test team finds in terms of keeping the development team busy fixing them. I would rather see teams learn more from their bugs by doing a causal analysis and try to improve their engineering systems to prevent the class of bugs in the future. In other words, learn from our mistakes at a team not just an individual level.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JR: As our software products become more complex I think we’ll begin to see more and more test specialization roles emerge like security expert or network protocol expert and the challenge will be to create a community to allow knowledge transfer so we can accelerate learning curves and mitigate risks in places where you wish you had an expert but instead have staffed the position with a smart but inexperienced person. The needs won’t be limited to technology but will also include domain expertise in fields such as healthcare, taxation, financials, or globalization. It is too much for one person to know everything and the costs of a mistake can be very high.

DDJ: Is there anything else you would like to say?
JR: I have a lot of respect for developers and other roles that are so critical to creating great software. That being said, I really love the testing discipline. Though very technical, it also allows me to tap into a creative side that really appeals to me. I think about the customer and am opinionated about what features should do and how they should integrate to make a compelling user experience. I enjoy the technologies and figuring out how to architect testability into our products. I get to write a lot of code focused on validating the product using the testability we designed into it. Finally, I get a bit of an ego rush in finding a bug. It’s something I try to keep respectful around the developers but when it’s just the test folks in the room I’ll let out my celebratory victory yell.

Posted by The Braidy Tester at 07:30 AM  Permalink |