THE BOOK OF TESTING

Thoughts From a Braidy Tester

August 2007

Anu Arora works in the Engineering Excellence group at Microsoft. Her interview contains pretty much everything I would typically write here, so rather than saying it twice I will hand you directly over to her. Here is what Anu has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
AA: The very first job I had as a aerospace engineer to work as a systems engineer in manufacturing helicopters. As a part of this job, I was involved in the ground testing of helicopters components and systems. I quickly realized that there was nothing more critical then the testing function in this industry. I also realized that I had an aptitude for it and enjoyed doing it.

My fascination for testing remained strong. Years later when an opportunity came through to make it as a career at Microsoft, I took it on.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
AA: The misuse of metrics! Whether it is code coverage, bug metrics, test coverage metrics, there is a lack of common understanding at what they really mean. In my career in software testing, it is interesting to see that sometimes people not only misinterpret metrics on their narrow understanding and then are adamant about their conclusions. I would like to remind folks that even the very well defined metrics are tools that provide us with data to help us make decisions. Metrics can’t replace commonsense and intuition in decision making, which comes with skills, knowledge and experience.

The other surprise was the constant struggle for testing to get their well deserved status. Today, when everything depends on software – mission critical, business critical and even life critical, you would assume that this discipline gets its due. However, it is undermined a lot in the industry. I think to some extent we are our worst enemies. We have compromised on the quality of testers we hire. We need to have testers who are qualified and passionate about the field and make the profession lucrative enough to retain people who are experienced.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
AA: Testers need to understand that testing is not just a lot of automation. Testers also need to understand that it is not the other extreme – banging the keyboard.

Testing is a highly skilled and sophisticated discipline and testers are the last line of defense for the customer. For the same reasons, the testers need to continually learn and grow; learn new ways to test, learn about the customer and market, learn to be always on the cutting edge of testing.

Developers need to understand that testers are not babysitters. Developers need to be responsible for their own code. Developers need to ensure what they write, works. In the testing language it is called unit testing. As a Test manager, I have been asked once in a while by the few ignorant ones; that if developers did all that, what will testers do? Testers can them move beyond unit testing to focus on integration, systems, scenario and user testing. These are absolutely critical from the customer point of view – the one whom we are building this product for.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
AA: I can see quite a few. The obvious ones are:

• Hiring qualified testers. This becomes difficult as most colleges don’t offer software testing in their CS curriculum.


• Retaining experienced people: Development and Program Manager are still the more glamorous careers to seek in the IT industry.

Not so obvious ones are:

• The shift from the typical product life cycles like Waterfall, V-shaped, etc. to Extreme Programming (XP) means a paradigm shift in testing from bug detection to bug prevention.


• The science of testing needs to continue to advance at a fast pace. With products getting more complex, customers demanding higher quality, and market demanding shorter time to ship, the testers will need the new ways of testing, methodologies, tools and process to aid them. There is a pressure on our fairly young discipline to become mature quickly.

DDJ: Is there anything else you would like to say?
AA: I have been in software test for nine years now. It was my passion for testing which brought me here and it is the variety, the breadth of experience which this discipline has given me which continues to excite and motivate me.

Are there challenges in testing today?

You betcha!

There are more myths about testing than anything else.

• That you can’t test-in quality.


• That high code coverage means high quality.


• That testing is there to just cause us delay to ship.


• Black box testing or white box testing is sufficient.

I enjoy educating people and breaking these myths. As a part of a small team of core instructors in testing for the entire company (in the engineering excellence team), I am responsible for the training and development of test leads and managers at Microsoft. I ensure that they have the learning resources and training to ship great products. I am also an instructor for the SDET (Software Design Engineer in Test) curriculum. It is both an honor and a huge responsibility. And, there is nothing else which I have enjoyed more in my career. My goal is that Microsoft continues to have world class testers who are prepared for the next set of challenges and have a huge role to play in shipping great (high quality) products.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
AA: What was the most defining moment in your testing career and why?

It was the space shuttle Columbia disaster on February 1, 2003. I was driving my kids for one of their weekend activities and I heard on the radio that my friend and fellow aerospace engineer from Punjab Engineering College, Kalpana Chawla, was killed along with the rest of the crew. As months went by, the shock settled in. I realized that it had impacted me in a big way. I started looking at my job as more then – I am a mere tester, my job is to just show the mirror. I became a quality advocate. I understood that mistakes and compromises can be very costly. I encouraged my test teams to bring evidence so that critical bugs can be fixed and never lost because we didn’t understand their magnitude. That meant more work for us, but it was worth it.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Joe McMahon has been involved in software development for almost thirty years. Back in 1979 he worked on satellite ground communication systems for NASA, eventually moving into IBM mainframe systems programming, Unix system administration, and earth science web applications. Then he moved to Yahoo!, where he has worked on test automation and is currently writing developer tools. He has done a bunch of Perl-related work as well, including enhancing the core test suite and documenting the Perl debugger. And now he's learning how to write audio software for OS X - which plays well with his other hobby of playing keyboards.

Here is what Joe has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JM: I fell into testing myself quite by accident. I was at OSCON, and Michael Schwern was doing a session on Test::More - a nice automated testing module for Perl programs. Long before the end of the session, I realized I was missing out on a great technique. I already knew that more automation meant more work done, but automating testing opened my eyes to the idea that not only could I get more done, but I could be more certain that it was going to work, and that once I'd fixed bugs, I could ensure they stayed fixed. Suddenly I had a lot more time. I felt free and empowered.

DDJ: How would you describe your testing philosophy?
JM: Any testing - as long as the results are dependable - is better than no testing. If you can demonstrate that testing will save them time and embarrassment, they will go from uninterested to enthusiastic.

Testing early is always better than testing later.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JM: The idea that not writing all your tests first somehow makes your testing less valid. The only real metric that applies here is that as long as you are testing, you are doing the right thing. Turning testing into something with things to feel bad about not doing is a sure way to discourage oneself. Do tests. They don't have to be pretty, or perfect. They only have to be dependable.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JM: For testers, that testing is always a search for the balance between the amount of effort expended on testing and the necessity to say "good enough, let's go with it', and that sometimes you're going to have to be unpopular and say "we haven't done enough yet" despite tight schedules. For developers, automated tests always have to pass. You can't get information from a test run if you have to decide whether or not failures are "significant". If they aren't, mark 'em as TODOs. If they are, well, fix 'em!

DDJ: What is the most interesting bug you have seen?
JM: My favorite one has to be Trey Harris's 500-mile email limit. The story's at http://www.ibiblio.org/harris/500milemail.html for anyone who hasn't read it. It's a beautiful description of assembling the data, testing, and getting that one idea that brings it together and solves the bug.

Posted by The Braidy Tester at 07:30 AM  Permalink |

Chris McMahon lives in the deepest depths of the Four Corners boonies, were he telecommutes in to Socialtext. Telecommuting from the boonies seem to have its rewards: Chris spends many of his lunch hours tubing the Animas river.

I first heard of Chris because some time back he was working to start a local market for custom software in his small town. Although he calls himself an enthusiastic beginner, he has presented at numerous conferences and written numerous articles for Better Software. Chris told me that he is the oldest customer of Watir; I am not sure whether he means old-as-in-the-first or old-as-in-age!

Here is what Chris has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
CM: Although I went to three universities on four different occasions, I have no college degree, and I spent my 20s as a semi-professional musician. I got tired of the road and I got my first ever 40 hours/week, 8-5 job at age 32. I was hired to write technical documentation.

I (luckily) shared an office with a senior developer named Pete. I got my first assignment and asked Pete how to run the software. He gave me a funny look and said "you're the first documentation person we've ever had that wants to run the software". In the course of my writing, I had a couple of technical questions and Pete showed me how to use the debugger. I had no idea at the time how unusual that was.

So I wrote my first assignment, using the debugger to check things when I wasn't sure how they were put together under the hood. I turned it in, and the next day the VP came storming into our office really, really mad. She said "Our software doesn't do this!" I said "Yes it does, watch this." I demonstrated what I had written. The VP's jaw dropped -- her first comment was "Oh, sh**!".

Testing takes a certain knack for being the critically important person who is never in the spotlight, working to make things go smoothly and working to prevent things from going haywire. When I was a musician, I was was never a singer or a songwriter or a lead guitarist, I was a bass player. Likewise, I don't want to be a developer or an architect or a CTO, I have a disposition suited for testing.

DDJ: What is the most interesting bug you have seen?
CM: I worked at a company that did data validation for 911 systems, so when you choke on that fishbone and dial 911, they know where your house is. This is a "perfect storm" bug...

The telephone companies add and split area codes pretty frequently. The 911 systems keep numbers around for both old and new area codes for a while before deleting the obsolete ones. The company had a routine little report we ran now and then for finding area codes ready to be deleted. We had someone run the report, but it was configured improperly, and instead of reporting obsolete area codes, it reported *all* area codes.

The sysadmin responsible for deleting area codes knew something was wrong, and resisted deleting the data. And resisted, and resisted, until he was outright ordered to delete the data. No one in management reviewed the report.

When this sort of data gets deleted, a loadable backup copy of each record is routinely recorded, just in case someone makes a mistake, and records have to be restored. And that happened in this case -- except that this was on an operating system where file sizes have to be declared when files are created. When the file is full, the OS returns an error message.

But the script that made the backup copies of the 911 records had never been tested -- it was a one-off created by a sysadmin directly on the production system. No one knew that the script never checked for error messages returned from the OS. So with this immense amount of data, soon the script filled the backup file. Then the script would write the backup record, fail, delete the original, write, fail, delete the original, thus destroying all copies of the records.

We deleted most of the 911 records for a particular Midwestern state that day. We restored them (which is another story altogether) -- but it was a very, very bad situation that could have been prevented at any of four or five different points in the process.

DDJ: How would you describe your testing philosophy?
CM: I know a number of testers who resist being labeled with the term "Quality Assurance". I used to resist it myself, but recently I've been coming back around, and now I'm starting to like the term "QA". Here's why:

I recently started working for a company called Socialtext. Socialtext is a mature startup that sells wiki software to Enterprise customers. The Wired Magazine "HowTo" wiki is the coolest thing Socialtext has done recently.

Businesses compete in three arenas: features; service; and quality. Startups by necessity focus on features first, service second, and quality third. Socialtext is just at the point where we have the features in place, service is maturing, and focus is shifting to quality.

So while testing is a big part of my job, another very important part of my job is to participate in the company culture as a an advocate for quality, to provide measurements that demonstrate quality (or lack of it), and to guide the ongoing internal discourse to discussions of quality.

I guess it's more "Quality Encouragement" than "Quality Assurance", but it's still a lot more than just testing.

DDJ: What do you think is the most important thing for a tester to know?
CM: I think that there are three technical skills that testers must have to be really successful. Testers really should be in a position to address: the filesystem; the database; and the network. Although some testers are subject matter experts, and some testers are process people, even non-technical testers are immensely more powerful if they learn how to manipulate files, speak SQL, and send information around over TCP/IP. It's not very hard to learn the basics, and the value these skills bring is immense.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
CM: Whether in manual tests or automated tests, I follow a very simple rule: do not test for blocking conditions. Do not test that a login succeeds; test for what happens after the login. Do not test that a page loads, test to manipulate elements in the page. Do not test for the presence of an input field; test that the field accepts input. The amount of time and money wasted on pointless tests is really shocking.

What's even more shocking is the number of people that disagree with me!

Posted by The Braidy Tester at 07:30 AM  Permalink |

Bill Buxton is a Principal Researcher with Microsoft Research. He is rather famous in the human-computer interaction research world. His book "Sketching User Experiences: Getting the Design Right and the Right Design”, published a few months ago, is one result of his thirty-plus years of investigating how people use technology. As you might infer from the title of his book, Bill is big on using simple methods to quickly prototype UI and other ideas. For example, if you were faced with the task of demoing a concept for improving the visibility of the mouse cursor as a presenter roams around a conference room, would you think about using Windows Movie Maker to stitch together a series of screen shots, and then coordinate the resulting movie into a live "demo"? Simple, no? Yet quite effective.

I mentioned that Bill is well-known in the HCI world. As in: his work has garnered him The Canadian Human-Computer Communications Society Achievement Award. He is well regarded in other worlds too: he was named the New Media Visionary Of The Year at the 2000 Canadian New Media Awards, and the Hollywood Reporter has named him one of the ten most influential innovators in the North American film industry, and the Ontario Horse Trials Association named him the 1996 Veteran Rider of the Year. Not exactly your typical computer research geek is he? Here is what Bill has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
BB: The first product I bought. I thought I was buying a product. What I was really doing was paying for the “privilege” of testing version n so that the producer could fix the design flaws in version n+1. Things haven’t changed much.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
BB: That testers are still spending time finding flaws that are there because of lack of design. It would be wonderful if they were there as quality control to ensure that a great design was executed greatly. That is almost never the case. Hence, they scramble to try and weave the proverbial silk purse out of the sow’s ear.

DDJ: What is the most interesting bug you have seen?
BB: Perfectly executed code that did exactly the wrong thing. Civil engineers do not get awards just because their bridges don’t fall down, and rather last for generations. That should be taken for granted. So should it be for software. That should be the starting point, not the goal. The objective, and the thing worthy of award, is a great design, beautifully executed. I’m still waiting.

DDJ: How would you describe your testing philosophy?
BB: Don’t waste the tester’s time on bugs that should never have occurred in the first place. Let their considerable talents be put to good and appropriate use.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
BB: The importance of design. That it is about getting the right design, as well as the design right. Just as a builder will likely come up with better results if working from great plans, so could it be with software. It never is.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
BB: Code reviews are not a substitute for what in the design world is known as the critique. We just try to refine code rather than seriously evaluate alternatives.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BB: To move a key part of design to the front end, prior to engineering, so that we know what we are going to build, that it can be built, how long it will take to be built, how much it will cost, who should build it, how it will be built, etc.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
BB: What we in the software industry should be asking ourselves is this: if our process is so great, and not needing a total overhaul, how is it that there is not a single company in the software industry that has a track record of producing new, as opposed to n+1 products? Our process sucks.

DDJ: Is there anything else you would like to say?
BB: Time to toss out the status quo. Current product development process does not work – extreme programming and agile development not withstanding.

Posted by The Braidy Tester at 07:30 AM  Permalink |