Five Questions With Bj Rollison

Bj Rollison studied cultural anthopology in college. (In Japan no less!) Now he is on Microsoft's Engineering Excellence team, where he trains testers, helps set testing policy, and works with teams across the company to get their testing to go where they want it to go today. Bj's GString Unicode string generator is handy for those times when you want to hit your app with some gnarly characters. If you're up for some strong opinions, head over to his blog I. M. Testy, which is appropriately named, for both senses of the word. Here's what Bj has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
BJR: My introduction into testing was actually building and testing various hardware configurations in the mid-80’s. I didn’t really think of it as testing then, it was more like “what pain am I going to have to go through today to get this thing to work!” The PC-AT machines were just starting to become more prevalent and peripheral devices for the early Intel x86 architecture presented untold challenges getting them to play well together. The industry lacked standards and the hardware changed every 6 months. Every hardware modification required resetting IRQs, DMA channels, and occasionally ‘flashing’ the BIOS chip. At that time a friend gave me some of the best advice I’ve ever received in my career in the computer industry. He said, “if you really want to know how a computer works, you need to learn how to program.” The hardware and coding skills provided strong foundations of how computers actually work, but it wasn’t really until coming to Microsoft that I faced the challenges of testing software in development. At first I asked myself, “how hard can this be?” Finding bugs was easy. But, it didn’t take me long to understand testing is so much more than simply finding bugs. Within weeks of starting I developed and ran the build verification test suite on 4 different language versions of Windows 95. I was responsible for determining if the weekly build was rejected and kicked back to the development team, released only for testing (usable but unstable), or released for self hosting (reasonably stable enough for the group manager and others to use for daily work). I had 30 minutes after each new build to determine its stability and provide qualified information to the team to justify my decision. Indecision was not an option. I learned very quickly that software testers must make hard decisions every day. Testers are responsible for making a myriad of critical decisions that could beneficially or adversely impact many people and ultimately the success of the project.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
BJR: One thing that has surprised me the most is how many people in the industry lack any sort of formal training in the profession. Anecdotal evidence suggests that less than 5% of testers have read more than 1 book on software testing, and Dorothy Graham stated that less than 10% of testers have received any type of formal training in the techniques and methods commonly used in our profession. Personally, I only read a few chapters from a book or two on software testing when I first started testing. It wasn’t until I became the Director of Test Training and had to design and develop training for new testers coming into Microsoft that I realized I didn’t really know as much about software testing as I thought I did. Books by Beizer, Myers, Jorgensen, Whittaker, Perry, Copeland, and Black gave me greater perspective, alternative viewpoints, and in-depth practical knowledge. Sure, I was successful in my early career finding bugs, leading teams, and managing testing projects. But, honestly it was a lot of trial and error. We were mostly successful, but success came at great cost both in individual burnout affecting team morale and attrition of our most experienced and senior testers, and company costs in terms of huge sustained engineering costs and reliance on superheroes to drive products out the door. Five or ten years ago the training opportunities were limited. Today, testers have multiple options to gain discipline knowledge and increase their technical skills to become better, more effective, more efficient, and quite simply more professional.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
BJR: I agree with Jerry Weinberg…mindlessly pounding on keyboards. I often refer to this as keyboard monkey testing, but some people have taken exception to that phrase, so I rephrased it to keyboard finger-dancing. Think about it this way, if you had to pay someone $5K for each test executed, I am sure if that person said “Let’s try this and see what happens,” your first response wouldn’t be “Cool…let’s try it dude,” it would probably be “explain precisely why you are executing this test, what hypothesis are you trying to prove or disprove and what test data are you going to use and how do you know that is the appropriate set of test data from all the possibilities?” When we think harder about the tests we design and execute, whether they are exploratory in nature or scripted or somewhere between, we could probably increase our effectiveness, efficiency, and confidence in the quality of our work.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BJR: The software industry is going through a cultural change right now, and the role of testing is near the nucleus of the change. Of course the changes are not entirely new. Fourteen years ago when I joined Microsoft the interview process included coding questions for testers. Virtually everyone was expected to be familiar with some coding language and reasonably proficient. About 1995 things started to change and many commercial software companies simply started throwing warm bodies banging away on keyboards to beat the bugs out of the software before it shipped. But, many things are changing in the industry whether we like it or not. Off-shoring and outsourcing are here to stay, and companies will primarily use those options for test execution. Changing development lifecycles such as Agile, XP, and TDD are renewing the emphasis on driving quality upstream. Essentially this makes developers more responsible for the quality of code they produce, and program managers more responsible for the quality of the requirements. From a business perspective this is good because it reduces cost. Some of the most prominent research going on these days involves static and dynamic analysis tools. These are testing tools, designed to identify defects earlier. This is not to suggest the role of the software tester is going away. In fact, these changes present new and exciting challenges to our profession. The testing discipline requires a great breadth of knowledge and demands a myriad of skills.

Testing is a highly skilled profession, and testers in the future will require even greater skills to remain valuable resources to software companies. Testers must be able to engage in both specification and code reviews. Analyze code coverage data and design tests to improve coverage. Design effective functional and non-functional tests from both a black box and white box perspective. And yes, testers should be able to automate their tests. Designing and writing effective automation tests is perhaps one of the hardest coding challenges I have faced. Frankly I think the days of the discipline filled with predominantly “black box” testers is quickly coming to an end, and those in the industry today must gain the appropriate knowledge and skills that are important for their company in order to remain competitive with their peers and with new people coming into the career field. Eventually, I see the role of the tester primarily focusing on test design and information analysis.

DDJ: Is there anything else you would like to say?
BJR: This is truly an exciting time to be in the field of software testing. There are tremendous changes on the horizon that will greatly affect the discipline far into the future. As testers, test leads, and test managers we are presented with a unique opportunity to change industry perceptions, to build credibility, and to break through the proverbial glass ceiling common to our discipline. There is a saying that it sometimes hurts to be on the leading edge of the sword. We can attempt to block the sword and continue to bang on keyboards in a vain attempt to test in quality, but we will be in the same predicament with the same problems we face today 5 years from now. Or, we could choose to seize the opportunity and fundamentally change the role of the software tester and impact how we test software in the future.

Posted by The Braidy Tester at 07:30 AM  Permalink