THE BOOK OF TESTING

Thoughts From a Braidy Tester

May 2008

Jason Blue Barile has been testing at Microsoft for some ten years now, working on products such as Internet Explorer and the Windows shell. Before that he lived the college life at Vanderbilt University, where he Research Assisted at the Intelligent Robotics Lab and worked on projects such as a theremin-playing robot. These days find Jason is Test Manager for Microsoft Visual Studio Team Foundation Server down South in North Carolina. I wonder if he's working on a testing robot....

Here is what Jason has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JBB: I've always been a tester at heart. I recall buying a book with 100 text-based games in BASIC when I was in 7th grade and spending hours on my Apple IIe debugging problems with code for a Star Trek game in the book. I recall being extremely satisfied each time I found a bug and marveling at how such a great book could be published with buggy source code. In high school, I ran a bulletin board system (BBS) and spent a lot of time working together with other sysops debugging problems with customizations to the software. In college, my professors were pretty hard core about our homework assignments working correctly, so my friend Carl and I would spend time adhoc testing each other's assignments before turning them in. I didn't have any formal training in software testing until I joined Microsoft, however. My first assignment was testing the APIs for ActiveDesktop in IE4.

I honestly never considered a career in software testing until my interview with Microsoft. I went into the interview hoping to become a developer, but the interviewer suggested testing given my passion for quality. Once she explained what Software Design Engineers in Test (SDETs at Microsoft) did all day, I was hooked in an instant!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JBB: You don't have to fix all the bugs :). When I started my testing career, I naively thought my developers would have to fix any and all bugs I turned up. I quickly learned a lot about prioritization, customer-focused testing, how to consider risk of regression when fixing bugs, etc.

DDJ: What is the most interesting bug you have seen?
JBB: Canned answer alert: they're all interesting! During Windows XP development, we had a bug where we had written some Accessibility code for the Shell. When the Shell was in right-to-left mode (e.g. for Arabic Windows), we forgot to flip the logic in the accessibility model. So when our test automation would ask the "close button" on a window frame where it was and then sent a click to those coordinates to close the window, the click would end up opening the frame's system menu instead. It was interesting to me from the perspective that most customers would never be exposed to this bug, but a subset of people using Accessibility tools on RTL builds might be completely blocked by it. So you shouldn't always just test for "most" customers.

DDJ: How would you describe your testing philosophy?
JBB: "Test Early / Test Often / Test Enough But Not Too Much" – Create a quality process that catches errors as early as possible, preferably before checkin, but also be realistic about the cost/benefit of testing at various stages. Find that sweet spot between not testing enough and testing too much (easier said than done!).

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JBB: I have a tough time choosing between debugging skills and communication skills, but if I have to pick one, I would go with the latter. Being able to explain software defects and their impact to the customer in simple terms is critical. Effectively summarizing the steps to reproduce the bug in as few words as possible can often make the difference between a bug being fixed or not – or at least being fixed correctly or not.

Debugging is a close second here. The more you know about the software you're testing and the failure paths of the code, the better. You'll be more effective in testing around bugs to look for related problems, and you'll also earn the respect of the developers you're working with. Respect helps build better professional relationships which leads to more and more effective communication with your team members, etc.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JBB: I can't think of anything off the top of my head. Perhaps "finding every bug" fits here. New testers (myself included, as I mentioned above) think you have to find every bug, no matter how obscure. What's really important is finding the bugs your customers will care about & making sure those get fixed first.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JBB: How to fit into an Agile world for many testers. How to increase efficiency and effectiveness.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you? What would you answer?
JBB: Why aren't more people interested in software testing as a career? I think people just don't understand how much fun software testing can be. There is a common misconception in the industry that testers can't be as technical as developers, aren't as creative, or are somehow "second class citizens". I know lots of people who used to be developers and got bored with constantly working on the same feature area for years on end. As a tester, you get exposure to new features and technologies all the time. You get a great breadth and depth of understanding of your product, and you're the one fighting for the customer (remind me who pays the bills again?). It's such a critical role in the software development process that it just doesn't make any sense to me why testers would ever feel like they're not important.

DDJ: Is there anything else you would like to say?
JBB: Thanks for this opportunity to share! I hope these interviews help inspire folks to consider software testing as their new career path!

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Some months ago I posted about CAST 2008, how it's not your typical conference, how gobs of Smart Testers will be attending, and how you probably should attend too.

The program is up now, and it's a doozy! Jerry Weinberg is presenting the opening keynote and also an all-day tutorial on communication and interaction models. Cem Kaner is keynoting about why testing checklists can be helpful even though test scripts often aren't. Michael Bolton, Jonathan Kohl, Scott Barber, and several other testers will explain parallels they see between testing and music, civil engineering, magic, improv, and labor room triage. And there's more!

Check out the program, then head over to the registration page and let them know you'll be attending. (Note: Early Bird registration ends 31 May 2008, so hurry over if you want to save a bit of money.)

Posted by The Braidy Tester at 07:30 AM  Permalink |

Elfriede Dustin has written numerous books and articles about testing and test automation. While I haven't read any of them myself, multiple someones evidently find her books useful as they have been translated into multiple languages and sell worldwide. She co-chairs the annual VERIFY conference, where one of the three presentation tracks focuses on (surprise!) test automation. She got her start in computers doing a different type of automation - using Wang computers to automate claims processing for the United States federal government. These days she is helping the US military automate their software testing, which seems a rather different challenge than the ones on which I work.

Here is what Elfriede has to say:

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
ED: Most surprising to me is that still often over 60% of the software development lifecycle is spent on software testing. For example, a most recent quote in Redmond Developer News states that "Industry analysts estimate that developers spend only about 20 percent of their time designing and coding ... the bulk of their time is spent on resolving application problems…" I have worked in various software development/testing environments, i.e. commercial and Government, and the testing challenges exist across the board, independent of the environment. Software testing techniques lag behind hardware testing techniques, for example, where use of tools is now regarded as mandatory. Today manual test generation for hardware is inconceivable; yet software testing often is still done manually in many companies. Software testing needs to advance to the same level as hardware testing; i.e. all software testing should be automated to allow for various efficiencies, such as software test reuse and repeatability.

DDJ: How would you describe your testing philosophy?
ED: My testing philosophy is to automate, automate, and automate. Manual testing is labor intensive, increasingly repetitive and thus error prone. It is archaic and cannot keep up with the software development advances. Despite the advances in development practices and tools, the goals of accelerating the rate at which systems can be delivered and reducing their costs cannot be met by simply writing software faster without comparable improvement in the practices and tools for testing the software. Software testing techniques are not keeping pace with software development advances. With the increased use of advanced software development tools, languages, and model driven architectures by software providers, software testing is rapidly becoming the "very longest and most expensive pole in the tent" when it comes to fielding new capabilities into production or to the masses. Automated testing is the answer. Automated testing tools and methodologies (such as the Automated Testing Lifecycle Methodology (ATLM) described in my book Automated Software Testing) have been developed and continue to emerge. Automated software quality tools growth is primarily due to businesses’ desire for higher quality, increased productivity, and faster time to market. Open source available automated testing tools are improving and STAF/STAX is one great example of a powerful open source automation framework.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
ED: The most important thing for testers (or anyone) to know is that testing requires skill and expertise. Software testing is a profession and should not be considered only a stepping stone into another department, for example into development. Often the misconception exists that anyone who is able to hold a clipboard can be a tester. I have written various books and articles on the skills required to be a tester (for a list of my books, see http://www.effectivesoftwaretesting.com/Books.aspx); i.e. to be an efficient software tester the person really has to enjoy testing, and not only see it as a temporary involuntary assignment; have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, to understand the technical intricacies of the application-under-test in order to be able to automate the testing efforts, b) understand the business, and c) never stop learning and improving on the testing efforts. Developers need to unit test their code and make automated unit testing part of their build processes, which results in the most successful development efforts.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
ED: The biggest challenge will be that testers and the test discipline will continue to play "catch-up" with development advances. With today’s processor cores tearing through billions of instructions per second and software builds of n-tier applications consisting of thousands of source files, computer hardware and software technologies have made many significant advances. Advances in development practices, methodologies, and tools which have been enablers in fielding phenomenal new, innovative, affordable, and timely information processing capabilities will continue to make headlines. In this existing paradigm of rapid development and edge of technology advancements, with a focus on reducing development time and creating re-usable capabilities, efficient and thorough testing of the software will be more critical than ever. Testing techniques and technologies will need to be able to keep up with the phenomenal advancements: Automated Software testing and self-testable components are part of the answer to this challenge.

DDJ: What else should I ask you?
ED: Where do you currently work and what excites you about your current job?

DDJ: What is your answer?
ED: I currently work at Innovative Defense Technologies. The exciting part about this company is that our goal is to improve software testing efforts across DOD and commercial markets, for example we are bringing automated testing solutions to mission critical systems at the Navy and soon to other parts of the DOD. IDT as a company is solely dedicated to improving software testing efforts across the board. We are working with edge of technology testing techniques and have some of the brightest leadership and developers assigned to the software automation tasks and efforts. IDT will change how testing gets done in the long run.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |