THE BOOK OF TESTING

Thoughts From a Braidy Tester

December 2007

Once upon a time Jared Quinert tested console games. Then he led teams of testers of console games. Then he built the testing department for a console game which never actually shipped. Currently he is spreading the context-driven testing love throughout the corporate world as a consultant with Aegeon, one of those Web 2.0 companies.

Here is what Jared has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JQ: I started as a games tester at the beginning of 1995, catching the tail-end of ROM-based games and the transition to CD-based PC and console game titles. Due to the expense of a product recall, those games needed to be rock-solid. Back in those days, our test approach was 100% exploratory, manual testing.

On my first real project, a bug report was thrust into my hand at some point, and I was told to wander around and ask the programmers what they had been working on and what they had changed in the build they had just given us. They would share information about things they thought were complex, risky, or otherwise. I would tell them what kinds of things our team was finding in the product.

After four or five months I was thrown onto a title as test lead, managing one other tester. Suddenly managers were wandering up to me and asking 'How's it going? When will we be done? Can we ship it?' Pretty quickly, I needed to figure out how to answer those questions, and thus I learned about developing coverage models to support exploratory test management.

The company I was at also developed the first third-party Windows game for Microsoft. Microsoft came to us wanting a heavily scripted test approach. Our test group wasn't even aware that people tried to test things by writing all their test ideas down. We definitely felt we weren't going to test *our* product that way. After initially rejecting the idea, I did begin to think about what problems they might be trying to solve with their approach. Some of the ideas were adapted and found their way into our regular testing to solve particular problems. So while it was great to get the testing grounding that I did, I'm also grateful that my view of things was challenged fairly early on in my career.

The big things my first job left with were:

• The idea of testing as a service role, and information provider. This came from the relationships we had with developers and producers.


• The idea that exploratory testing, done well, can deliver quality far above what I've generally seen using heavily scripted testing approaches.


• The knowledge that if talented people do exploratory testing day in, day out, they get really good at it.


• The idea that face-to-face communication is critical. We moved offices, were split up across multiple floors of the building, and bad things started to happen.


• The knowledge that having skilled testers allows you to travel much lighter.


• The idea that testers have to tell a compelling and realistic story when reporting bugs. We were often representing the concerns of important people who didn't have a voice on our team: Gamers of different skill levels, with different preferences, publishers and support people.

My understanding now is that not all games testing groups rely on smaller, skilled teams, so I could quite easily have come out of a games testing role with a very different view of testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JQ: The big surprise was how useful the skills I developed as a games tester were when I moved to corporate environments. I think this was largely due to the testing philosophy instilled by my first role. I was also surprised by how much worse I perceived the testing in those environments to be. Using exploratory approaches, it was really easy to find bugs that had been lurking in systems for years. Given the potential financial losses involved in many of these systems, I had expected to encounter testing even more rigorous than my games experiences.

Testing for internal audiences I find is a much vaguer experience than consumer product testing. The politics are more complex, and problems are solved in different ways. Learning to be an effective tester in different environments is a huge, probably career-long task.

DDJ: What is the most interesting bug you have seen?
JQ: The most interesting games-testing bugs were usually needle-in-haystack kinds of problems. I remember one of the first bugs I found, which was a crash that happened only after playing seventy percent through a game, then applying an obscure button combination while attacking an enemy which only existed in this one room. I then surprised myself by playing the game back through to that section and reproducing the problem first time. I think that as the programmers later explained to me why it had crashed, I connected the dots and the bug value of null pointers was suddenly burned into my brain.

I think another thing that never ceases to surprise me about good testers, and that's how much bad luck seems to follow them around when they're testing. Bugs seemingly fall into their laps. While this may seem like an unfair advantage, I think it needs to be backed up by good investigative skills and the tester's ability to pull the pieces together and retrace their steps.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JQ: I see a few challenges. Firstly, that business right now seems particularly keen to reduce the cost of testing. Secondly, developers are becoming interested in testing, and getting better at it. At least in my part of the woods, much of what passes for testing is heavily requirements-based, and fairly clerical. Between outsourcing and enthusiastic developer test automation efforts, traditional tester roles are going to be under increased pressure to deliver value. If we want to have successful, satisfying careers as testers, I think we are going to need to skill up and figure out how to keep adding (enough) value.

The third key challenge is figuring out how to get some experimentation into the workplace and push the boundaries of what we know about testing, in order to test better. Persuading businesses to take on appropriate business risk for an appropriate reward seems like a hard sell from my view on the ground.

DDJ: Is there anything else you would like to say?
JQ: I've sort of undertaken a mission to change the testing landscape here in Melbourne, Australia. It's a slow process, but a community is growing. If anyone feels that they can contribute, I'd love to hear from them!

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Kevin Beto was, once upon a time, a professional wrestler. And a cheerleader. Not at the same time, he claims. Seven years ago he traded bashing other people for bashing applications at Microsoft. Which is probably a good thing, as he is currently a test lead on Microsoft Office PerformancePoint, and bashing heads is generally frowned upon by all the best people managers today.

Here is what Kevin has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KB: My first introduction to professional testing was at Microsoft. I had a CS degree and only had development experience, and didn’t realize I was interviewing for a test internship here at Microsoft until half way through the interview when I absolutely got my butt kicked by the “Test a toaster” question. I thought it was the most boring thing to do in the world when given a handful of manual test cases to run, until my mentor introduced me to a tool called “Genesis” that did genetic algorithm testing. I spent only a week implementing that tool for a zip code conversion library that we had and found all sorts of really juicy bugs. It would just run overnight and find bugs which would be sitting on my computer in the morning for me to pump into our bug reporting database. That is really what changed my opinion of testing from people who just click buttons until something breaks to a very challenging engineering discipline.

Ever since then, all I think about is how I can do things more innovatively or efficiently. Now I love the manual stuff too because I see so much opportunity in finding ways to make that faster and easier.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KB: I was surprised how complex and exciting testing can be. I continually run into fresh challenges that stretch me, and always find some new way for me to grow myself professionally.

DDJ: What is the most interesting bug you have seen?
KB: We ran into a really interesting threading issue about a year ago. Our automated BVTs suddenly started randomly failing in the middle of execution. It was never the same test case, but it was always the same error. After about a month of investigation, arguing with development, proving older builds worked, and newsgroup postings, we were able to track down the issue to a product bug. Turns out the product was creating a whole bunch of threads, which overflowed the thread pool and exposed a really nasty synchronization issue if you had the right amount of data in your system. This of course would repro on the multiprocessor lab machines but never on the single processor development machines.

DDJ: How would you describe your testing philosophy?
KB: I really like to plan for the long term when doing testing. I am willing to cut short term goals down in size if I know that one or two releases from now I’ll be able to accomplish orders of magnitude more than I am doing today. I invest heavily in automation and long term projects (model based testing, etc.) that will allow us to at least keep, if not exceed, pace as the scope of the product grows without needing to add headcount.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
KB: That we’re all on the same team and working toward the same goal. It is very easy to draw a battle line and line up on opposing sides and make everything a contentious issue, but the really high performing teams I’ve been on had an unprecedented amount of cooperation between development and testing. Once that last feature is checked in, everyone should be looking for bugs. Test should be focusing on finding the new issues, and dev should be focusing on finding places where known bugs have close relatives in other parts of the code.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KB: Finding every last little bug, especially on new products. At some point, you need to label the quality as “good enough” and get it out in front of customers. You may ship buggy software, but given how agile we can be with the advent of ubiquitous connectivity and the internet it is easy to make patches to fix almost anything that makes it past the test team and gets found by a customer. Having that customer feedback to improve and adjust your testing practices and coverage in my opinion is much more efficient than trying to come up with it on your own.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
KB: With the introduction of managed code that enables rapid development practices, and testability taking a front seat in many platforms (IE, .NET Framework, Avalon, etc.), I believe we’re right at the brink of a huge wave of technical innovation in the testing world with both the tools and techniques we use to do our job. The challenge is going to be collaborating/leveraging existing solutions that are out there and avoiding the sometimes irrepressible urge to reinvent the wheel.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

James Hamilton started out as an auto mechanic in Ottawa, Ontario, almost thirty years ago. After a few years of that he decided to become a code jockey instead and so pursued a couple CS degrees. With those in hand he went off to IBM, where he worked on projects like IBM's first C++ language compiler and garnered a fistful of awards. From there he moved to Microsoft, where he spent stints on SQL Server, Windows NT, and Exchange, and is now helping the Windows Live Platform Services team build a high scale services platform. On top of all that he speaks at a plethora of conferences, serves on a multitude of industry committees, and publishes papers on a regular basis.

Here is what James has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JH: My first experience with testing was in building systems as part of my undergraduate degree in computer science. I did well with these assignments but it wasn’t by being a better developer. University Computer Science programs are full of good developers. My secret weapons were: 1) build componentized systems that can be independently tested at each component, 2) build simple automated testing systems that can drive all component tests and overall system functional tests, and 3) having made it possible to test and easy to run all the tests, do it any time for anything no matter how small the change. Even a comment. The last point is probably more succinctly summarized as having patience and follow through. It’s perhaps the easiest statement to make, the most boring attribute to see in print and yet it’s one of the most important attribute of both good developers and good testers. In all forms of engineering, the first 90% is easy. Getting something running reasonable well and mostly correct is easy. It’s the last 10% that separates great software from frustrating software and it’s that last 10% that separates good developer and testers from the average.

Going back to the three secret weapons, it’s ironic that I was able to do considerably better than average on development work at university by investing in testing rather than in development. By spending more on testing upfront and automating the tests, I was able evolve systems more quickly and efficiently and the resultant systems were often more correct. If you agree with me that great testing is one of the keys to building complex systems, why is it that so few universities actually teach testing?

My first job after university was leading an Ada compiler test team at IBM. My initial observation was that there was huge opportunity for improvement and I’ve since learned that this is true industry-wide. In my opinion, if you want to fundamentally improve software quality and the speed with which it’s produced, it makes most sense to go where the biggest opportunities for improvement lie. Software testing and design and development for testability is, in my view, the easiest and best way to improve the current state of the industry.

DDJ: What is the most interesting bug you have seen?
JH: Over the years, every software release on which I’ve been involved has had some truly difficult bugs. One of the most interesting was just prior to shipping DB2 V2. I was the Unix architect on DB2 and this was our first Unix port of the DB2 UDB database management system. This was a super important release from a fairly new team and the company was watching. Like all software projects, we were late and there was considerable pressure to ship. We had one remaining serious bug. A very rare index corruption bug that only showed up under load. This is exactly where many software projects hurt themselves and their customers by bending to the pressure to ship and, by so doing, failing over the long haul. Janet Perna who lead the DB2 development team knew that a new database just entering the Unix database market would be quickly irrelevant if customers encountered even a rare data corruption bug. She was 100% right and this is the hardest decision that development leaders have to make. Years later Paul Flessner who led the SQL Server team said, in a tough shiproom discussion: if we ship this bug, customers will still remember 10 years from now whereas if we’re a month late, it’ll be forgotten before the end of year.

Geoff Peddle, one of the best engineers on the DB2 team had been chasing this index corruption bug for weeks. As the release got later, more and more engineers got involved in tracking this bug but the weeks dragged on. This bug always showed up in high scale, long haul stress runs but was nearly impossible to reproduce. And, when index corruption was detected, the actual issue that corrupted the index may have happened hours ago. We tried all sorts of tricks to catch the problem as it was happening including hooking the lowest level of the system where pages are written out and putting full page integrity checks in place. Nothing would reproduce the problem at its occurrence but, once or twice a day, the long haul stress machines would bump into it.

We eventually found it – I wish I could recall who figured it out (drop me a note if you remember). The problem was a cache firmware problem. One broadly distributed hard disk model had a firmware cache problem. On rare occurrences under heavy load, a cache error would show up where a write would be written to cache but not to disk. On subsequent reads, the old page contents from the previous page generation are returned directly from disk. To make things a bit more fun to debug, seconds later the disk firmware would force the dirty page from the cache to disk restoring overall database integrity. This means when it comes time to debug the data corruption, every page is now perfect which led us all to be convinced that it was a DB2 problem since the on-disk contents were always correct.

Since the problem was in non-flashable disk firmware, the fix was to change the device driver for this manufacturer’s disk line to disable the cache for affected disk drives. The unfortunate side effect of this fix is that, for the thousands of customers who uses these disks, when they did a device driver upgrade, their disk caches were shut off which will substantially negatively impact most workloads.

We did the right thing to delay shipping to get this problem resolved since the faulty device was the most common disk in use on the RS/6000 which was our primary target platform. Had we shipped on time, customers would have remembered that bug and hated us for the data corruption problems that followed for the subsequent decade.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JH: The clear division between test and development that many in the industry still advocate today simply doesn’t work. In healthy teams, testers are developers by all measures. They may not be directly coding on the product or service but they are writing code and they need to be as capable as the rest of the engineering team. Large test teams are a symptom of a problem rather than a way to solve it. I’m not saying we shouldn’t have engineers who focus solely on testing. My point is that these folks need to have all the skills of developers. They need to be just as capable and as well compensated as any other member of the engineering team.

Just as testers are developers, developers need to test as well. All developers need to be responsible for unit testing. By “responsible” I mean that, when code is checked in, it should be functionally correct. A few companies have operated this way in the past where developers were both expected to fully unit test AND there was a considerable social and professional stigma associated with checking in code that later showed multiple functional problems. At these companies, development included unit test and developers were held accountable for the both the quality and timeliness of what they produce. At companies where developers write code and testers write tests, developers end up being mostly responsible for code quantity and schedule and considerably less accountable for quality. This is bad for the developers and does terrible things to the products, rewarding what we need least: lots of nearly correct code.

If developers are responsible for unit test, the code will be organized such that it’s easy to test, the overall quality will be better and the code produced will be smaller, tighter, and more correct.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
JH: Testers need to take a customer perspective with a developers knowledge. They need to know the system as a whole better than developers and they need to understand and represent the users and administrators of the system. They need to keep the overall project focused on ease of use and ease of administration and they should be actively using the product or service in every way practical.

Developers need to design for testability. Systems that are hard to test are hard to ship. Developers need to own unit test and design systems that can be effectively tested through automated tests below the UI level. Years ago the semi-conductor industry responded to exploding complexity by employing design for testability and by including test circuitry in the actual shipped component. We need to do exactly the same thing with software. We should be designing systems that are easy to test and we should be including self-test code in the production systems. Just as modern CPUs include elaborate self-test circuitry, production software systems should spend resources implementing self test.

Systems where substantial resources are spent on self test are easy to evolve going forward, show far less software entropy and generally have lower service costs.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JH: Testing doesn’t scale. Software complexity continues to increase and is leading to non-linear test team growth. I’ve worked with teams that have more testers than developers and yet their product quality was still not under control. Bigger test teams usually end up doing more of what doesn’t scale and they just continue to grow. Eventually someone determines that to control costs, we should offshore the testing work. They feel great and may even be rewarded for taking 1/3 the cost out of the testing effort but it’s bad engineering piled on bad engineering. The base problem is that testing doesn’t scale and growing test teams to cope with the non-linearly scaling complexity doesn’t work. Sending large, ineffective teams off-shore does reduce costs temporarily but they are still ineffective and they still can’t scale.

The only way to address the exploding complexity of large software systems is to design them for testability. Developers need to view “done” as having written the code, written the unit tests necessary to establish that the code works, and to ensure that the tests pass. Development shops that view “done” as code complete are doomed to be late and ship low quality software. Developers should be held accountable for the number of bugs found in their code after it has been checked in and declared done. For the most part, this isn’t tracked and often isn’t even noticed.

The biggest challenge for the test discipline over the next 5 years is to focus on this core problem: testing doesn’t scale. Test teams need to educate the development groups in design for testability and the production systems need to include self test code. Testers need to push this part of their jobs to development. Successful test teams will show the leadership to push unit test to development, to automate all they do, and to be part of all design and overall project decision. A healthy test team will be a small, high talent organization that is a full part of all product decisions. The test team needs to represent the users and administrators of their product and to ensure that what ships actually works for the target audience.

Successful test teams will be the ones that innovate in finding ways to allow test to scale. Successful test teams may get smaller than many current generation test teams but the overall engineering talent density on these teams may actually rise. To continue to be successful, the industry as a whole needs to address the test scaling problem on large software systems.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |