Five Questions With Evan Goldring

Evan Goldring is a test architect at Microsoft. Today, anyway; before that he was a Group Manager (i.e., he ran an entire product group and managed the people who managed the various disciplines). Before that he was a Dev Manager (which is to say he managed the people who managed the developers for a product group). Before that he was a tester. And before that he was a developer. And before that he was a tester. And before that...you get the picture, I imagine: Evan has switched between development and testing a lot! He says he keeps coming back to testing for the challenge and creativity. Which I guess means development is rote and boring. <g/>

Here is what Evan has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
EG: Like most (all?) developers my first introduction was not because I wanted to test but because my programs had bugs in them. I know I fixed millions of bugs before this, and clearly had to do some amount of testing to find those bugs, but my intro into real formal testing – testing others' code - happened as the result of a bug.

I was visiting with a family friend who was already a professional programmer. I was a Senior in High School and they were helping me wrestle with the question of whether I should major in Computer Science or Computer Engineering. I knew it’d be one of the two, but I really didn’t understand the difference. At some point, he left me to play with some budgeting software he had created for his dad to use. When he came back in the room I told him I found a bug in it. One of the values was coming up negative when it shouldn’t have. He asked if I knew why that was happening and I thought it was pretty obvious and quickly said "Yeah, I’m guessing you are using float instead of a double and I’m overflowing it." What I didn’t realize was that he wasn’t quizzing me, his mind was racing through mathematical and algorithmic errors he could have made but came up empty. It turned out that everywhere else, a float was fine – the values never got that large. But, the total had to be a double.

It was because of that that I landed an internship with his company the next summer where my job was to test desktop software to do mainframe session management. Without being asked, I found automation software to use, devised my own way to measure my "coverage" of the system, and left them with about 80 pages of documentation on the test methodology, instructions on how to run the tests, and a full gap analysis of what I didn’t cover as well as fully documented automation scripts.

They had thought I’d just test the software by using it (ad-hoc testing) and log bugs all summer long. I was under the assumption that clearly this being a professional environment "just" doing ad-hoc testing wouldn’t get me a return job the next summer so I had to step up my game, but I had no idea what that meant so I sort of made it up. Back then, I couldn’t just search the Internet for "software testing". I just thought – if I was in charge what would I have to see from a college intern to be impressed.

I don’t know if that is a statement about how bad they were or how good I was (the company made solid software, made a lot of money, and was bought out by one of the largest software firms for a crazy sum of money before the tech boom so I don’t think they were that bad), but it was a very positive experience for me because I ended up being somewhat self-taught in testing vs. being ramped up by others. By deriving a good test strategy on my own I could better appreciate what others had done in the field by the time I found articles and books on software testing. It’s sort of like learning calculus through memorization vs. derivation.

DDJ: What is the most interesting bug you have seen?
EG: One time I was doing memory leak testing on a streaming media system and found that, under the right conditions we would find an exponentially increasing memory leak happening, but at the same time a similarly decreasing CPU utilization pattern. Since all of us (dev, test, and PM) had inherited this giant mess of code with little documentation, none of us could figure out why we’de see a pattern like this – it just didn’t make sense. Luckily, I had proven myself so the typical route of questioning the tests didn’t happen :).

We set out to figure out what was going on and we found that there was some exponential backoff code that kicked in when the connectivity pattern was in a certain state. That explained the CPU graph, but only further made the memory leak graph unexplainable – you’d expect to see the opposite effect. Well, it turned out that memory wasn’t cleaned up in each case and the algorithm was trying "harder" each time, but waiting longer in between. "Harder" meant doing more work with more connection attempts that weren’t getting cleaned up.

None of this could have been found without collecting a ton of data and graphing them. A picture (created by automated tests), in this case, was worth 1,000 hours of manual testing!

DDJ: How would you describe your testing philosophy?
EG: There are two elements to my philosophy. The first is "Always have a balanced, but wide, test arsenal". If you over invest, you can miss stuff and not be well equipped with the right debugging and diagnostic tools when a certain bug comes around. You have to know when to use automation, know when to use manual testing, and know how to do manual testing efficiently and effectively with "directed ad-hoc" testing. The first thing I do when faced with testing a new component – no matter how big or small – is to create the big picture of what tools I need in my tool box to enable me to attack it from every angle, telling me everything about the system I need. These tools could be anything from software hooks, to well placed “printf’s” (asserts, exception handling, etc), to diagnostic tools, to linear functional regression tests. Once I have the big picture, the second piece kicks in – customer focused testing.

By customer-focused testing, I mean thinking about what sort of bugs each piece, or the depth of each piece, of the picture would find and what functionality it would verify. I prioritize based on the impact that bug would have on the customer. I’ve seen a lot of people, especially junior testers, get excited by finding a corner-case exception but ignore a layout issue as "fit and finish". In reality, the exception could be an extreme corner case – severe, but hardly ever hit – but the layout bug could chop off important text or cause extra mouse clicks in a usage flow that a user goes through 50 times a day. Likewise, looking at how actionable error messages are. Performance (or at least perceived performance) and other characteristics of the software are important to measure. This is how I prioritize the big picture I create and ensure that I build the test tools that will find the most important – from the customers point of view – issues. In other words, I really like to test with the end user in mind.

Once I have the prioritization, I go through and create all the cases as manual ones. That way, at the end of the project, I’ve at least got a record of all the testing I wanted to do and have a backup plan in my back pocket (using an army of manual testers to hit everything). That usually takes on the order of days. Then I start automating, building the tools, etc. Aside from a great backup-plan, this approach has a secondary benefit – you understand the testing you want to do with the automation. The best automation is born from manual processes that were repeated so often that they are well understood and could be replicated easily, without loosing potency, by code. I’m not saying you have to run each test case 100s of times before you can automate, but by at least laying out all the tests as manual ones, it requires you to think through the steps and run through them a few times to make sure they are repeatable and predictable. This small tweak – creating all the tests as manual ones before automating – has saved my teams time and time again either because we can deeply Plan B if some automation doesn’t happen to fit in the schedule or because we thought through the test cases first which saves us from going in the wrong direction and throwing away a lot of work. This is one of the highest return on investment activities you can do on a project.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
EG: Testing the parts vs. testing the whole. As I said in my philosophy, you need to find the right balance. I’ve used products, as I’m sure everyone has, that are just dreadful in one way or another. If you talk to individual testers, they will claim their piece is only contributing to that metric in a positive way. However, the sum of the parts is much smaller than the characteristics of the whole. Performance is a great example of this. If you aren’t looking at both the product-wide and the "nuts and bolts" testing of all the pieces that make up the flow of that scenario, you won’t know what the user's experience will be, nor will you know where to look if that scenario’s performance doesn’t meet the bar. I’ve seen many large-scale projects where too much emphasis was put on the parts, but little ownership of the gaps between them existed. This, at least, has been in some of the projects I’ve been exposed to, and this isn’t a comment specific to Microsoft.

This isn’t a direct answer to the question, but I think too much emphasis is often placed on component level testing and I think some of that should be traded for a look at the whole to strike that balance. Using the second part of my philosophy, I’d prioritize against the severity and likelihood of a customer coming across a bug to know what to trade off.

It is admittedly a tricky business. Is a data loss bug that one person out of 10,000 will hit more important to go after than a more benign issue that will cause one out of 1,000 to have to reboot every 10th time they run the program? Is truncated text on a localized build with only 5,000 projected users more important than every users needing 2 more mouse clicks to do a common task more important? At the end of the day, the answer is going to lie in what is motivating and driving the business so the answer could be different in each company, in each product. But, you need to really understand those elements well. Not all bugs are created equal.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EG: We need to learn how to drive quality - in every sense of the word.

Software Engineering is unlike most other Engineering disciplines, in that the notion of "quality" extends far beyond the traditional bounds. Ease of use, doing what customers want and need from their software, easy and unobtrusive upgrades and patches, etc. are all necessary. You find these notions in other engineering disciplines, but few have all of them like software engineering.

I think our challenge is in learning from these other disciplines so it doesn’t take us 100+ years to become as refined as they are in their approach to quality assurance, yet still enable quick turnaround on new features, and driving quality across the entire product development cycle, not as a bolted-on activity that happens near the end. Again, this is not a reflection of Microsoft, but rather what I’ve seen on average across the industry.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink