THE BOOK OF TESTING

Thoughts From a Braidy Tester

March 2007

Jon Bach is Manager for Corporate Intellect at Quardev, a software testing and technical communications consulting firm in Seattle, Washington. I first met Jon at STAR East last year, where he tested an application before a live audience of hundreds of testers while his brother James provided running color commentary. Jon is president of this year’s Conference for the Association for Software Testing, and he runs Seattle's own QASIG which discusses all things testing. He'll be presenting at STAR East again this year, this time talking about the Session-Based Exploratory Testing method he and his brother invented some years ago. Here is what Jon has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JB: It was an entry-level data processing job entering commercial real estate listings for an online database, but I was always running into bugs. There were only 8 people in the company, and no testers. I reported what I found to the developer, who seemed to appreciate that. That felt good. What also felt good was when he said it saved him a lot of headaches because he was tired of the tech support guy forwarding him all of the calls, when instead he could focus on fixing things before the customers ever found them. Call it my Myers-Briggs type (ESFJ), but I liked being of service. I also realized I was in service to the customers who might never need to call tech support if I found all of the bugs! It became a treasure hunt, a gauntlet thrown at my feet to see how many problems I could find. In my mind, I wasn’t making software better, I was saving lives! To this day (12 years later), it still feels that way – like I’m a secret service agent. The fact that the software seems to get better when my bugs are fixed is just icing on the cake.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JB: What surprises me is that the people I respect most in this business – the people with years and years of varied project experience, speaking at international conferences, writing a million articles – they all bring something new to the table every year. You’d think after a certain point, they would know all there is to know and would become complacent or boring or pompous. I know many “experts” do go that route, but the people I respect always want to contribute something new and of value. That shouldn’t surprise me, but it always does. Maybe I’m surprised because these experts are so much like me? Could it be that they, too, are still getting over that “impostor syndrome” ever since their first testing job? My first official testing job (where I was actually hired as a tester, not a data processor) was with Microsoft, where suddenly I was working next to people with CS degrees. The fact that Microsoft took a chance on me in light of what they had to pick from was a surprise in itself. But I found good bugs for them, and they didn’t fire me, so I knew I must have been doing something right. What they called my “energy and enthusiasm” may actually just have been an eagerness to show them the different ways I could be qualified for the job I already had – to not let them down. Surprisingly, the people I respect most in this business seem to have the same attitude.

DDJ: What is the most interesting bug you have seen?
JB: I have to confess that in all the projects I’ve been on, the most interesting one is something I saw at a testing workshop, courtesy of instructor Grigori Melnik from the University of Calgary. You can try it right now. Run notepad.exe and type "this app can break". Save it as test.txt, close it, then open it again. You might see that it now shows all rectangles, as if the font couldn’t be loaded. Ironically, it was part of Grigori’s talk titled “Test-Infecting Future Engineers” to a group of 20 seasoned testing instructors invited to the Workshop on the Teaching of Software Testing and it wound up infecting all of us with curiosity and energy. My brother James seemed to be the most “infected” and decided to take on the challenge of investigating this behavior with Grigori looking on. He recorded his process in a 20-minute video during lunch that day. The cool part for me was that unlike everyone else who got rectangles when they opened the file, I got what looked like Simplified Chinese characters! One of the people in the workshop happened to be Chinese, so I asked him what it said. He said “it says garbage, nothing”. That was a big clue that inspired my own investigation, but James took this to a deeper level. It inspired me to think, how much of the day should testers spend investigating cool bugs? Could it be a waste of time when a programmer already has enough information?

DDJ: How would you describe your testing philosophy?
JB: As a service provider (working in an outsource lab), I have to find things that are of more value to my clients than my competitors can. I learned that from Robert Sabourin at Amibug, who works very hard to instill it in his students at McGill University and demonstrate it in his consulting. But value-driven testing is not just a business philosophy, it’s also a personal ethic. I’m self-critical and the projects I’m on are usually short and have small budgets, so that combination of factors leads me to have the mantras: “is this test valuable”, “is my strategy likely to reveal something valuable?” and “am *I* valuable”? It starts with a healthy sense of paranoia and a desire to learn more than just what the mission is for the projects I’m given. What about the Bigger-Picture Mission, for example? Once I know the mission, sometimes it’s useful to ask, “why is that my mission?” to begin to see a larger system in place that may impact my testing approach and techniques. Seeing systems is the most important philosophical thing I’ve learned in testing. I have my brother to thank for that – mainly for introducing me to Peter Senge’s book – The Fifth Discipline. Senge didn’t call this out by name, but his book was the first time I learned the difference between recognizing faults and recognizing failures. If you don’t consider system interactions or consider the software as a whole system of features, you may end up reporting failures, not their underlying causes (faults). It’s not the end of the world if you don’t find the fault – sometimes there’s not enough time -- but I prefer to build up my credibility bank account as soon as I can with the people to which I’m in service by investigating as much as I can into why the bug happened. The contract I want with them is: “don’t waste my time having me test things you already know are broken, and I won’t waste your time giving you bugs that are shallow.” Since I’m interested in quickly winning autonomy and trust with my clients, knowing the difference between fault and failure is one huge way to do that (and to me, it’s the most apparent skill that differentiates novice testers from veteran ones).

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JB: Let me answer this like I did the other day when speaking to college seniors: “Be cautious, critical, curious.” Technology can be taught much easier (in my experience) than how to make someone curious and engaged, yet have a healthy sense of self-critique. If they are curious and engaged, you can teach them all kinds of ways to harness that. I also say “take a risk and ask a question.” This is something I see the most seasoned testing experts do – well, at least the people *I* call experts. It takes practice (and courage at first) to ask questions, because when you ask a question, you’re making yourself vulnerable. You’re admitting you don’t know something and you risk someone thinking less of you, not more. That’s why there are a lot of “experts” out there that never seem to ask a single honest question. I assume that’s because they think they can’t afford to have people think less of them. But the people I respect most not only ask questions, they ask all *types* of questions, inspiring me to look just as hard at where answers may be hiding as where bugs may be hiding. After all, a test is just a question in disguise, and tests are my business. Now on the dev side, they should know that testers are trying hard to be of service. When testers celebrate a bug, they are not celebrating a developer’s mistakes, they’re celebrating the fact that the software did not fool them this time. This is an important celebration because the software so often fools testers into thinking it is working most of the time! All said and done, I think programmers should test for a day, and testers should program something at least once in their careers.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have considered which platforms to include in and which to omit from your test matrix. The set of supported platforms tends to vary widely across contexts - a consumer application will likely have a different set of supported platforms than does an enterprise line of business application. Even if you officially support only a few specific platforms, it can be useful to understand what will happen if your application is installed or executed on other platforms. Platforms to consider include:

• All supported versions of Windows; at a minimum: Windows XP SP2, Windows XP SP<latest>, Windows Server 2003 SP<latest>, Windows Vista SP<latest>
• Apple OS X.<latest>
• Your favorite distribution of Linux
• Your favorite flavor of Unix
• 32-bit version of the operating system running on 32-bit hardware
• 32-bit version of the operating system running on 64-bit hardware
• 64-bit version of the operating system running on 64-bit hardware
• The various SKUs of the operating system
• Interoperability between different SKUs of the operating system
• Interoperability between different operating systems (e.g., using a Windows Vista machine to open a file which is stored on a Linux network share)
• All supported browsers and browser versions; at a minimum: Internet Explorer 6, Internet Explorer 7, Opera, FireFox
• With and without anti-virus software installed
• With and without firewall software installed

Also peruse the Windows Logo requirements. Even if you aren't pursuing logo compliance (or your application doesn't run on Windows!) these are a useful jumping off point for brainstorming test cases!

Posted by The Braidy Tester at 07:30 AM  Permalink |

Robert Straavaldson is the best tester I know. He is the famous Rob from my Hallmarks of a Great Tester speech. His developers on one project pooled their funds to have their dev manager to take Robert to lunch - for multiple weeks in a row! - so that he would stop logging bugs for an hour each day. On another project he was asked to stop referring to bugs as "cool", and to stop logging bugs during bug triage meetings. Robert even tested this interview - note that he boundary tested my "Five Questions" by answering six of them! Here is what Robert has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
RS: My first introduction to testing was in the late 1980’s with AutoLISP. I remember spending many hours writing scripts that I hoped would either fail miserably or crash the host application. Good times!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
RS: I think the most surprising change is how I look at things in my everyday life. I’ve gone from being mostly oblivious to my surroundings to noticing everything. I’ve also noticed that I’ve gone from being the person that expects everything to work and not really caring how it works to trying to figure out how it works and how I can make it to fail.

DDJ: What is the most interesting bug you have seen?
RS: There’s been so many over the last 7+ years and to keep it simple I’d have to say any bug that I was able to make my developer ask “Why would you do something like that?” after reading the bug report.

DDJ: How would you describe your testing philosophy?
RS: 1) Building a strong relationship with your developer and PM builds a better product.
2) Finding bugs is a good thing! Get excited about it!
3) Everything matters.
4) If you have to do anything more than once and it’s inexpensive – automate it.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
RS: I think this goes for both test/dev and it’s that testing isn’t something that you do at the end of the product cycle. Testing should always be done in measureable increments (by feature) over the life of the project.

DDJ: Is there anything else you would like to say?
RS: Since a very young age I’ve had the “I wonder what’ll happen if I do this?” trait. It turns out this is the type of thinking that all testers need. Being curious and devious at the same time is a good thing.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have looked at each and every file that makes up your application, for they are chock full of information which is often ignored. And we all know what happens when things are ignored - bugs appear! I remember one bug bash where a developer chalked up over fifty bugs simply by going through this list!

• Verify the version number of each file is correct.
• Verify the assembly version number of each managed assembly is correct. Generally the assembly version number and the file version number should match. They are specified via different mechanisms, however, and must explicitly be kept in sync.
• Verify the copyright information for each file is correct.
• Verify each file is digitally signed - or not, as appropriate. Verify its digital signature is correct.
• Verify each file is installed to the correct location. (Also see the Setup YANDY.)
• Verify you know the dependencies of each file. Verify each dependency is either installed by your setup or guaranteed to be on the machine.
• Check what happens when each file - and each of its dependencies - is missing.
• Check each file for recognizable words and phrases. Determine whether each word or phrase you find is something you are comfortable with your customers seeing.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Bj Rollison studied cultural anthopology in college. (In Japan no less!) Now he is on Microsoft's Engineering Excellence team, where he trains testers, helps set testing policy, and works with teams across the company to get their testing to go where they want it to go today. Bj's GString Unicode string generator is handy for those times when you want to hit your app with some gnarly characters. If you're up for some strong opinions, head over to his blog I. M. Testy, which is appropriately named, for both senses of the word. Here's what Bj has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
BJR: My introduction into testing was actually building and testing various hardware configurations in the mid-80’s. I didn’t really think of it as testing then, it was more like “what pain am I going to have to go through today to get this thing to work!” The PC-AT machines were just starting to become more prevalent and peripheral devices for the early Intel x86 architecture presented untold challenges getting them to play well together. The industry lacked standards and the hardware changed every 6 months. Every hardware modification required resetting IRQs, DMA channels, and occasionally ‘flashing’ the BIOS chip. At that time a friend gave me some of the best advice I’ve ever received in my career in the computer industry. He said, “if you really want to know how a computer works, you need to learn how to program.” The hardware and coding skills provided strong foundations of how computers actually work, but it wasn’t really until coming to Microsoft that I faced the challenges of testing software in development. At first I asked myself, “how hard can this be?” Finding bugs was easy. But, it didn’t take me long to understand testing is so much more than simply finding bugs. Within weeks of starting I developed and ran the build verification test suite on 4 different language versions of Windows 95. I was responsible for determining if the weekly build was rejected and kicked back to the development team, released only for testing (usable but unstable), or released for self hosting (reasonably stable enough for the group manager and others to use for daily work). I had 30 minutes after each new build to determine its stability and provide qualified information to the team to justify my decision. Indecision was not an option. I learned very quickly that software testers must make hard decisions every day. Testers are responsible for making a myriad of critical decisions that could beneficially or adversely impact many people and ultimately the success of the project.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
BJR: One thing that has surprised me the most is how many people in the industry lack any sort of formal training in the profession. Anecdotal evidence suggests that less than 5% of testers have read more than 1 book on software testing, and Dorothy Graham stated that less than 10% of testers have received any type of formal training in the techniques and methods commonly used in our profession. Personally, I only read a few chapters from a book or two on software testing when I first started testing. It wasn’t until I became the Director of Test Training and had to design and develop training for new testers coming into Microsoft that I realized I didn’t really know as much about software testing as I thought I did. Books by Beizer, Myers, Jorgensen, Whittaker, Perry, Copeland, and Black gave me greater perspective, alternative viewpoints, and in-depth practical knowledge. Sure, I was successful in my early career finding bugs, leading teams, and managing testing projects. But, honestly it was a lot of trial and error. We were mostly successful, but success came at great cost both in individual burnout affecting team morale and attrition of our most experienced and senior testers, and company costs in terms of huge sustained engineering costs and reliance on superheroes to drive products out the door. Five or ten years ago the training opportunities were limited. Today, testers have multiple options to gain discipline knowledge and increase their technical skills to become better, more effective, more efficient, and quite simply more professional.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
BJR: I agree with Jerry Weinberg…mindlessly pounding on keyboards. I often refer to this as keyboard monkey testing, but some people have taken exception to that phrase, so I rephrased it to keyboard finger-dancing. Think about it this way, if you had to pay someone $5K for each test executed, I am sure if that person said “Let’s try this and see what happens,” your first response wouldn’t be “Cool…let’s try it dude,” it would probably be “explain precisely why you are executing this test, what hypothesis are you trying to prove or disprove and what test data are you going to use and how do you know that is the appropriate set of test data from all the possibilities?” When we think harder about the tests we design and execute, whether they are exploratory in nature or scripted or somewhere between, we could probably increase our effectiveness, efficiency, and confidence in the quality of our work.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BJR: The software industry is going through a cultural change right now, and the role of testing is near the nucleus of the change. Of course the changes are not entirely new. Fourteen years ago when I joined Microsoft the interview process included coding questions for testers. Virtually everyone was expected to be familiar with some coding language and reasonably proficient. About 1995 things started to change and many commercial software companies simply started throwing warm bodies banging away on keyboards to beat the bugs out of the software before it shipped. But, many things are changing in the industry whether we like it or not. Off-shoring and outsourcing are here to stay, and companies will primarily use those options for test execution. Changing development lifecycles such as Agile, XP, and TDD are renewing the emphasis on driving quality upstream. Essentially this makes developers more responsible for the quality of code they produce, and program managers more responsible for the quality of the requirements. From a business perspective this is good because it reduces cost. Some of the most prominent research going on these days involves static and dynamic analysis tools. These are testing tools, designed to identify defects earlier. This is not to suggest the role of the software tester is going away. In fact, these changes present new and exciting challenges to our profession. The testing discipline requires a great breadth of knowledge and demands a myriad of skills.

Testing is a highly skilled profession, and testers in the future will require even greater skills to remain valuable resources to software companies. Testers must be able to engage in both specification and code reviews. Analyze code coverage data and design tests to improve coverage. Design effective functional and non-functional tests from both a black box and white box perspective. And yes, testers should be able to automate their tests. Designing and writing effective automation tests is perhaps one of the hardest coding challenges I have faced. Frankly I think the days of the discipline filled with predominantly “black box” testers is quickly coming to an end, and those in the industry today must gain the appropriate knowledge and skills that are important for their company in order to remain competitive with their peers and with new people coming into the career field. Eventually, I see the role of the tester primarily focusing on test design and information analysis.

DDJ: Is there anything else you would like to say?
BJR: This is truly an exciting time to be in the field of software testing. There are tremendous changes on the horizon that will greatly affect the discipline far into the future. As testers, test leads, and test managers we are presented with a unique opportunity to change industry perceptions, to build credibility, and to break through the proverbial glass ceiling common to our discipline. There is a saying that it sometimes hurts to be on the leading edge of the sword. We can attempt to block the sword and continue to bang on keyboards in a vain attempt to test in quality, but we will be in the same predicament with the same problems we face today 5 years from now. Or, we could choose to seize the opportunity and fundamentally change the role of the software tester and impact how we test software in the future.

Posted by The Braidy Tester at 07:30 AM  Permalink |