THE BOOK OF TESTING

Thoughts From a Braidy Tester

June 2008

Jim Bullock has been building software for twenty-five years. He has been writing articles and books which engage your mind for almost as long. And I am sure he has been an interesting person for at least as long.

I remember the first time I met Jim in person. I figured we'd talk for an hour or so and then I'd be on my way. Instead, we talked for over four hours. We stopped then only because we each had supper engagements!

One reason Jim and I talked from one meal clear through to the next is that both of us want to understand most everything. Jim has a knack for asking questions which uncover other questions which uncover other questions and so on until he uncovers the root of whatever matter is being discussed. I discovered this helps him be an excellent lunch companion. A plethora of companies have discovered this also helps him move companies from floundering to fabulous.

Jim pays attention to who does what, when. He calls this Conscious Development. I believe he would join me in also calling it fun. Here is what Jim has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JB: I learned to develop software supporting University research labs. Everything was novel, we needed it to work and suspected it wouldn't. So, I was first exposed to software testing with real questions, mostly about my own work, and a personal interest in the answers. That environment was also hugely influential to me - people creating experiments to learn something they wanted to know. We tested our software from the same POV. In testing software I think what you want to know comes first. From that you invent experiments and the gear to support them. We ask software testers to do experiments on software for us because they are crafty and find interesting things.

The first independent software testers I worked with tested my team's heat pump control software at Carrier Corporation. It was both disconcerting and comforting having other people poking at our stuff. Most of the problems found we'd correctly implemented the wrong thing. After that we got testers looking at plans, requirements, designs and so on. You can test anything, and it's worth it. Something happens in people's brains when they think in terms of interrogating the thing in hand. I also learned the value of independent eyes. With those systems we developers were sure what the system should do, and we were wrong. It turns out your own understanding blinds you.

Ever since that experience at Carrier, I try to get ignorant eyes on every work product. To start with, independent testing is a kind of ignorant eyes not blinded by what you thought as you did the work. With code I organize development in a Lean / FDD hybrid with strong gating. Then, every integration gate includes a developer who didn't work on the change and a tester. As new eyes, those reviewers are really a test case, too, standing in for maintainers who come along later. A successful system is going to outlive the original team, so "somebody new" and "recovering history" are two use cases every system will experience. Why not test for them? (Actually the Ingres engineering VP talks about this in the WSJ business technology blog - first week of June, 2008).

I organize testing the same way – Lean / FDD work flow with strong gating and lots of "ignorant eyes" along the way. With testing the extra eyes are especially helpful since testing only gets action when someone else accepts the results. The extra eyes test your delivery as well as your results, and how it plays for someone else, not just for you.

I was first responsible for independent testing while running the implementation of a 3-city, distributed IT system on a new product. I oversaw the vendor's development and testing as part of our product acceptance, and managed our functional testing and specialty tests like performance and data conversion. From this I learned that independent testing also flushes out configuration management & build, requirements management, and development practices – really any kind of communication. Since then, I always make a "process error" defect category for things like these. I also learned that in evaluating your product other people assess how you did the work – explicitly or on the sly. So, it is good to have a compelling, transparent story of what you did and why. Then test your story.

Since then, in PM and development roles (including life cycle tools development, which is fun) I co-created a testing consulting practice in the mid-1990s serving multi-tier enterprise IT development projects. After that I spun up several QA / test teams for both web site companies and products. Each company or project is a little different because different systems have different questions. "Correctness" sure, but correct in what sense? Under what circumstances? By the way, what are the project and company constraints on the testing we can do? Somewhere in there people decided I was a tester. I'm not so sure. I mostly align the team and tools to the needs at hand. Sometimes I get to do testing myself, mostly around architectural risks or performance.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JB: I'm surprised by people focusing on ritual vs. results. I've seen testers insist on doing stuff that finds nothing. I've seen developers insist that "You must test this way" when something else is finding lots of valuable results. I've seen testers and developers ignore impassioned questions about the software because their preferred techniques do something else. I've seen testers & developers ignore the performance that the project or organization needs from testing.

I'm baffled. If testing one way finds nothing, try something else. If testing another way finds things that matter, well, that result is all the justification you need for the technique. If there's something you need to know about the system you have, figure out how to discover it. I think testing starts with questions: "What do we want to do better?" and "What would we like to know more about?" followed by: "How do we find that out?" and "Is the testing we are doing helping?"

I sometimes find testers, developers, PMs & others griping about each other, too. "Those guys take too long." or "Those guys build stuff we can't test." Sometimes people act like those other guys can fail but somehow we'll still be OK. Maybe it's because I bounce between roles but I don't get that. It's not a lifeboat where the guy who falls out is in trouble while we're OK. It's a human chain where if anyone is in trouble we all are. We need these people. Measure your contribution in terms of how you create opportunities for others to contribute in turn.

When I'm PM-guy or running development, I tell testers: "Here's what I could use from you." Then I ask: "What do you need from me to be able to do your job?" and "What else can you offer?" The better they do their job, the easier my job, so why wouldn't I? When I'm doing testing I focus on the kind of information development, sponsors and Software Process Improvement (SPI) need about the system, and the kind of performance in doing testing that will help them succeed. Techniques follow from that.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
JB: There's a lot of arguing from position that I don't get - "schools of testing", methods and movements (the current loud one is "Agile"), and individual techniques ("unit tests" and "model driven testing" for two.) I don't get it. The pragmatic people I know go: "Interesting . . ." to anything that does some good, and "Well, duh" to the limitations. Professionals have a kind of informed open-mindedness.

The other day on his blog Keith Braithwaite quoted an old-school paper by D. L. Parnas. He points to an article by Michael Feathers, that refers to something from a while back called "Clean Room Software Development." Clean Room Development is nearly the opposite of "Agile" in many of its choices, and it works, too. (Maybe if you turn the knobs down to 0 that's also interesting.) Testing in Clean Room Development works stochastically from the outside exploring a formal model of the state space the system will encounter. So Feathers and Keith, both of whom would probably call themselves "Agilists", were both looking for interesting ideas, observing what happened, and seeking to understand. Isn't that the testing mindset? Feathers concludes that techniques that work encourage reflecting on how your stuff is supposed to work. That's the right way to do it. Arguing from position is for amateurs.

Arguing from a position in testing is most virulent around techniques. For example "All tests must be automated" is silly, while declining to automate tests you sensibly can is just as dumb. For all the good of the current emphasis on "developer testing" - and by the way, when was developers not testing their work OK? - some folks get narrow-minded about it. Developer testing by itself is sufficient? What about those ignorant eyes? Or more subtly, does xUnit give you a design technique called TDD, an automated regression suite making refactoring safer or a harness for building interesting tests that operate at APIs? I pick "all of the above", and "do what makes sense", but you'll find people arguing from position that it is one or the other.

The thinking seems to get shallow when advocacy sets in. What's left seems like partial descriptions to me. Sensible testing does something like "V model" if you look at it that way, but isn't just that. Clean Room Development's stochastic testing is "black box" but not "black box" in the way many people mean. It's incredibly disciplined and thoughtful. I think a professional tester is able to pick among the different approaches and tools out there. I think a lot of people have a good and useful partial description of testing software. A great example of this is Cem Kaner's paper on 100 different kinds of test coverage. What, 99 of those folks were being silly? Or maybe each thinks their kind of coverage makes sense in their situation. Maybe it does. Now, you pick the kinds of coverage measure that make sense in your situation.

There's a better way:

• The risks and priorities of your system and project point to questions you
  want to answer better.


• The tools & methods that fit come from those questions.


• Whatever you discover is valuable information.


• Do the most useful thing you can, in the situation at hand.

DDJ: Going meta (to channel Jerry Weinberg), what else should I ask you?
JB: What doesn't get enough attention and should?

DDJ: What would you answer?
JB: SPI doesn't get enough attention, and the little it gets is wrong. If you're going to have testing or QA (they're different), have ongoing SPI too. Too many teams keep making the same old mistakes then finding them in the same old way. Everybody is busy but it's dysfunctional, literally codependent. Each group creates an artificial need for the other. What would testing do if development eliminated their usual ways of screwing up? What would development do if testing started finding new categories of problems? What is either group doing about the problems that got past the way they work already? If the other guys learn and change how they work, you'll have to change how you work too, and that's uncomfortable. With testing or QA you have data to drive that if you are willing.

I saw this at a small company client where I was Interim QA & Test guy, then Interim Engineering VP. Doing a maintenance release addressing a dozen known, critical problems testing found 60+ more, equally critical defects and bounced every attempted repair at least once. The regression rate of changes was steady over four months. After I moved to Engineering, it was three months of shipping both 50+ repairs / month and new development before testing found their first problem - same guys, same lab vs. the same product. Five months and counting after finding that problem testing had yet to find another. Meanwhile, via investigations to resolve failures in the field engineering identified several new common kinds of faults in the system and set about removing them across the whole product - hundreds, still with no apparent regression.

The difference was data-driven SPI. First, in test we used the defects we discovered to get more crafty about exercising the system. If configuration variations were failing, we'd try more configuration things until that stopped being fruitful. Same with techniques or approaches to testing. Meanwhile, engineering kept working the same way they had. Later, in engineering we had 4 months of data telling us our most common kinds of mistakes - code management screw-ups, problems ill-understood before repair and design-free changes. (That's in DeMarco's "The Deadline" and it fits with my experience. Once you straighten out the life cycle and tool chain at all, the vast majority of "bugs" are under-design.) So, we straightened out code management then added "explain the problem" and "explain how your design works" to change gating and our results got better.

By the way, I don't believe we had no regression over those eight-plus months. I believe we didn't notice any in engineering and didn't find any in testing. That made me nervous. Somebody should have been thinking of other ways to poke at the system, to find other stuff. In test / QA the team and I had identified 10 kinds of testing we could throw at that system - each "kind" a type of fault and a technique to get at them. We'd executed on one, expanding the coverage over 10x. We'd started applying two others when I transitioned to Engineering, and that's where it stopped. Testing wasn't finding much new because they weren't trying anything new, while Engineering was getting smarter about how to work.

For SPI to work we have to get away from thinking that "SPI = progress toward the one true way." I see too much process-ritual in developing & testing software - stuff that Must Be Done(tm). Better to look at what's working and not working for us, here and now. What does our experience tell us? The way we do testing or development is also a test – an experiment in building software a particular way. If what we try doesn't work, try something else. We also need everyone involved in SPI to look first at their performance, then at the performance of development as a whole. Too often "SPI" means "scolding other people." Too often "SPI" means "sub-optimize the whole to optimize my part for me." I've seen testers and developers do that, and it's silly.

In the example I gave we mobilized resources to let people work better. Testing got permission to go after additional questions and ways to poke at the system, as long as they were fruitful. In engineering "strong gating" really gave management cover to engineers taking the time to think through what they were doing. Effective SPI is really about noticing what you could do better, then allocating some resources to doing that. There is no scolding in effective SPI.

DDJ: Is there anything else you would like to say?
JB: Personally, I emphasize the human interactions around testing and the value testing delivers in the organization. I may have a selection bias, but in my experience effective testing and software development depends on interactions first, processes & practices second and specific techniques last. Fix the former and the latter correct themselves.

For example, my "Big Book of Perfect Software Testing" - a lightning talk and one page of notes - talks about the value testing delivers. Twice that I know of it's been included verbatim in other people's training, which is quite cool. My "Calculating The Value of Testing" in STQE (now Better Software) talks about the value of testing as a business function from executives' POV as sponsors.

On human interactions, Brain Branagan and I presented "The Three r's of Software Testing" about ways for testers to keep their heads in the game. Stresses in a project or organization come out in testing, in part because testing done right injects reality. "The Three r's" is about working with that. There's an article that goes with the presentation that I probably should publish. Or look at "Chinese Contracts" on the AYE Conference site. That's about agreements between people working together, like testers, developers and PMs.

Oh, yeah - hire me. I say I do "conscious development." It's amazing what happens when you pay attention to what you are doing when you develop software. If you like what you are hearing contact me - contract or direct / permanent for the right gig in Seattle.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Microsoft Test Architect Keith Stobie scales mountains for a living. Currently he is helping Microsoft's Protocol Engineering Team construct model-based tests for Microsoft's wire protocols. A perfect use of model-based testing, you might say; also a project which will take a while to summit. Keith has also worked with the Live Search and Windows Communication Foundation teams, where he undertook similar ventures. He was also an active member of the Web Services Interoperability (WS-I)'s Test Working Group, where he (surprise!) created tools for testing conformance to WS-I profiles. Most all of Keith's work for the last quarter-century, in fact, has involved testing distributed systems.

Keith scales mountains when he's not at work too, albeit ones of a rather more physical nature, like Washington State's Mt. Rainier and Mt. Adams.

Here is what Keith has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
KS: It was so long ago, I couldn’t concretely say. In college I spent my last two summers as a programmer intern for GE on their network management software. I frequently encountered bugs in their tools. I was discouraged to learn that we internal users were lowest priority to get a fix - after customer reported bugs and after internal QA/Test reported bugs. Ah to be a tester – my bugs might get fixed sooner.

At the end of the second summer while contemplating my future full time career, my manager suggested I consider Quality Assurance and Test because "you think differently". For one, I expected the product to be able to do whatever the manuals said! (ha ha ha). Most successful developers I know are exceedingly skilled at working around problems to make progress. Reporting the problem is secondary to getting their job done. For me, I run straight into the problem and want it fixed!

Many developers avoid problems by using tried and true (supposedly) examples or existing code - thus avoiding the pitfalls of unexplored areas. I assume I can code from reading the manual. If the manual doesn’t say there is a limit, I presume there is none.

Consequently, when I went interviewing after graduating, I listed on my cover letter I would consider software QA/Testing in addition to software development. This is still relatively rare today. I took an offer from Tandem Computers whose trademark was NonStop computers. They understood quality and appreciated testing - that helped a lot early in my career.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
KS: I suppose the lack of any theoretical foundation, especially when I started. Academia has only scratched the surface about providing a sound basis for software testing. Early debates raged around carefully directed testing and random testing (and continue today). We are still learning when random testing (e.g. Fuzz testing) is most advantageous and when structured testing (e.g. boundary value analysis) helps most. Similarly the early work on code coverage metrics relating to block, branch, path coverage and similar criteria was revealing, but not very conclusive. The advent of combinatorics still has some researchers doing fundamentally bad assumptions from a practical testing point of view. On the other hand methods like test prioritization based on code coverage have worked better than I might have expected.

DDJ: How would you describe your testing philosophy?
KS: I like the context driven school of testing. You need to have many approaches in your toolbox from which to choose. No single approach, tool, etc. applies all the time. However, I borrow from all the "schools" of testing as part of my work. I sometimes do rigorous testing (Analytical school) using Model Based Testing to show all state transitions are covered. I’m currently using Factory school for the 300 contractors we have building protocol test suites. I’ve just finished my second batch of reviewer training with emphasis on process and acting as gate keeper (Quality school) for the quality of test suites produced by our contractors.

DDJ: What do you think is the most important thing for a tester to know?
KS: Finding bugs is not the job. Providing information to assess release readiness is. Finding bugs is important. The reason a philosophy of bug finding is good Is because otherwise too often testers drop into a rut of doing merely confirmatory testing - which only partially assesses release readiness. I’ve also seen testers go to the other extreme and only think about tests which will probably find bugs – and ignore whether the product actually basically works.

DDJ: To do?
KS: Besides enjoy their work, they need to be constantly learning and becoming better educated. I am constantly astounded that so many testers that consider themselves professional or career testers have never even read a single book on testing! Most of the best and most senior testers I know love testing, just like I do.

DDJ: For developers to know and do about testing?
KS: I think test first or test driven development is great and more developers need to become "test infected". The most important thing for developers to know about testers is that testers don’t exist to clean up their mess. Good developers know this.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
KS: I’ve been lambasting code coverage for quite some time (see my ACM Queue article Too Darn Big to Test). Tools already exist that can automatically get you 100% code coverage. Of course they may not verify a single result, but they execute all of the code. What we need, but have no automated way to measure today, is verification coverage. I’ve required 70% code coverage from developer test suites. Sometimes they get the coverage, but the tests are laughable because there is no verification.

Good test suites have high code coverage, but high code coverage doesn’t necessarily mean a good test suite. Code coverage also says nothing about the equally important aspect of state coverage.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |

Edward Miller isn't the scientist who helped design and develop the Corona reconnaissance satellites. Nor is he the Edward Miller who rode with Jesse James. He is however the founder and CEO of Software Research, Inc., purveyors of web and client/server testing tools. Edward organized the first Florida Software Testing Workshop thirty years ago and has been part of the software testing community ever since. Between publishing articles in IEEE and ACM, organizing the QualityWeek conferences, and building the aforementioned tools, Edward has been supporting testing since before testing was cool.

Here is what Edward has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
EFM: Probably my first introduction to test was when I was working on writing programs to test student programs on an IBM 7094. Well, "trying to write" may be more accurate. That was before I learned what "undecidable" meant!

Later, in a much more serious vein, for a time in the early 1970's I worked in technology that involved attempting to verify quality in terminal guidance for interceptor missiles that actually might have had nuclear release authority for their nuclear warheads intended for anti-ICBM defense. It was such a different era -- when the Soviet Union as a threat was taken very, very seriously -- and missile defense was a serious, serious business.

And, while it was very challenging work it was also kind of frustrating because of the enormous technical difficulty.

One of the most difficult things for me to do as I recall was to explain to high ranking military types -- Generals with two or three or more stars! -- what ACTUALLY was going to be involved in confirming the quality, reliability and integrity of some of that missile-borne software.

By a curious twist, however, the extreme difficulty that we revealed in trying to confirm the software that was doing that kind of guidance and control work led, I'm sure only in small part and in concert with work done by a lot of other teams, to the USA signing on to the ABM treaty. That was a good thing, for sure!

Then later, in the Reagan era, "Star Wars" popped up and again was used to foment a kind of win for the Western Alliance with the demise of the Soviet Union.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
EFM: In a word, how really hard it is to test applications automatically.

For example, back in 2000, we were near the end of the client-server test tool cycle. Our product line, TestWorks, had been a success, but we knew things were changing. So, we looked into the crystal ball to try to see the future and -- no big surprise here, not rocket science -- the future we saw was "the web."

So we said, OK, if you have a web browser enabled application, what is the best way to test it? And the answer was pretty easy, and really obvious: test it from a browser.

That was the easy part, defining the outside concept. Just test the web application from the browser and you're done.

Some seven years later we now have released eValid V8, an IE clone browser that can test ANY kind of web browser application including websites, web services, e-commerce applications, etc. Because the test functionality is all inside the eValid browser the overhead for performing functions is very low, the fidelity of what is done is very high, and the ability to do tests reliably -- even when the web pages change around a lot -- is very good.

But this isn't a product pitch at all. The real point here is that we found out -- in many cases the hard way! -- that it is a lot more complicated to do a good job of automated testing of a web application than you'd think.

Many times I've said to myself, "if we knew then how complicated it was going to be to get this product going we might not have done it." Maybe we benefited from a kind of technical blindness to the underlying complexity of web applications, but that's water over the dam. eValid is a built engine and it functions pretty much as we intended and hoped.

You never really know until you try to build functionality into a web browser to allow you to test those functions just exactly what's actually going on "under the hood"... until you jump under the hood and try to get control of it! That's when the fun begins, and when things that are obvious turn out to be very, very subtle and difficult!

Believe me, it has been a real challenge. As we often say around here, "We don't get bored...".

But it turned out for the best. Our technology now has a U.S. Patent (with more patents applied for), and there is very strong interest in the technical community in the browser-based approach to web testing applications.

DDJ: How would you describe your testing philosophy?
EFM: There's an old line that goes like this; "In testing good news is good news, bad news is good news. The only news in software testing that's bad news is NO news!"

In other words, you have to REALLY pay attention to what you're doing and what you see and what you observe! Everything may turn out to be important, and it takes a very sharp eye and very careful thinking to do testing well. And, if all your tests pass you learn nothing. Worse, if all your tests fail, you learn even less.

Software testing is, in essence, structured experimentation -- a very disciplined kind of experimentation. Certain stimulii produce certain responses in the thing you're testing -- the UUT (the Unit Under Test) -- and the goal is to somehow characterize whether those responses are OK or not, relative to some understanding of what the object is supposed to do.

It's great when there is a "formal specification" you can test against, but most of the time that's unavailable and instead you have to perform your experiments against something of lesser context and detail. And in that case, instinct, experience, good judgement, and -- to be honest -- quite a bit of luck is needed to get a good result.

Not every test has to produce a failure to be valuable. I think too often people focus too much on testing to reveal failures rather than testing to confirm operation. Of course, finding and documenting anomalies is important, but that's maybe only 1% of the whole picture of a "mostly OK application."

Applying these ideas in the eValid technology brought much of the basics of testing into very sharp focus. In the context of testing web browser enabled applications -- which typically are extremely complex and rich with functionality -- even the slightest errors can cause real havoc.

DDJ: What is the most interesting bug you have seen?
EFM: It seems to me you have to parse 'interesting' in two dimensions: intricacy and value. Intricacy as in "how hard to detect", and value as in "how much impact does the defect have?"

Peter Neumann's "Risks" publishes the biggies. Almost always human goofs that have big impact.

But here is a subtle problem we found that might have taken years to find and might have cost users multiple millions.

A hardware product we were working on performs a kind of virtualization of web access -- in the virtual private network space. There is software in the device that scans HTML passages that are coming over the wire and has to manipulate them in a non-invasive way. What we found as the result of applying some very tricky test suites to the device was what looked at first like just a simple LAN-transmission anomaly.

But it was a consistent and repeatable anomaly.

We finally isolated the problem down to a programming error in one of the HTML processing modules, and we created from that guesswork a set of simple HTML passages that were passed through incorrectly every time.

This was very hard to detect -- in fact, it was a problem with the product that had been there for many years. But as the web-page sophistication level increased, this particular defect had increasingly negative effect. We never finalized the total value lost to the client, but it was certainly multiple millions.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
EFM: The quick answer: The Web!

The web is in many ways a real mess, yet it has enough redundancy and resiliency so that it thrives anyway. But far too many websites and web applications are vulnerable to all kinds of problems: pages that don't work, sloppy JavaScript that eats performance, new technologies that have (ahem, shall we say) "rough edges". (The guilty will need no accuser here!)

These days it's way beyond "broken links" -- those are there of course -- the issues are at the level of very powerful applications that just don't work reliably enough, or consistently enough. Ask yourself, did you see some website (or web application) in the last week that turned you off? Of course you did; it's unavoidable.

But if I had to list the web quality issues they'd come down to these:

• Web 2.0. The implications of the Web 2.0 revolution in collaborative websites is going to create some very tough quality and testing challenges. You have dynamic content, global asynchrony, and a raft of other tough issues to deal with.


• AJAX. The hard part about AJAX is that the better it gets the more "Asynchronous" it gets, and doing QA/QC/Testing on something that isn't terribly consistent in when it does what it does is daunting.


• Web Development Environments. New systems, like Google's GWT, the Morfik system, and many others like them, make it easy to create big, robust, websites. If you don't do things right, you ALSO wind up with websites that are nearly impossible to do any serious testing on.


• Service Level Assurance. How do you successfully assure that a web application, which could be hosted "in the cloud" or anywhere, really is working correctly? This requires Rich Internet Application monitoring, which is a kind of repetitive functional test process.


• Competitive Webiste Comparison. As web-based commerce moves closer and closer to being the main mechanism of "doing business" it's inevitable that firms will need to figure out how their website compares with the competition. Right now that's almost a total unknown! The methods we know about are effective, but they are TOO dependent on human interpretation. Like everything in computing it's gotta be automated.

[See my Table Of Contents post for more details about this interview series.]

Posted by The Braidy Tester at 07:30 AM  Permalink |