Five Questions With Payson Hall

You may be familiar with Payson Hall if you read Better Software magazine, or Computerworld, or the Journal of the Institute of Chartered Financial Analysts of India. Or maybe your company was one of the innumerable hardware and software systems integration projects he has consulted on and executed on over the past twenty-five years. Or maybe you have participated in the classes and seminars on systems integration and project management and risk management Payson has been leading at various seminars and conferences for many of those years.

Systems integration, project management, and risk management may not seem to have much to do with testing. If so, take a moment to look at your product through those points of view, and then see if you don't approach testing your product differently. I find Payson's thinkings help me understand these different points of view, which in turn helps me find and prevent issues I likely would have missed otherwise. Which seems all goodness to me!

Here is what Payson has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
PH: My first introduction to testing was to be hired as the part time tester for a commercial product about to be released by a two person company. I was pretty green. I hadn't finished my computer science degree yet, and my notion of testing was "try to find a systematic way to beat on the keys and see if the product breaks". Four years later, as a product manager at the same company (now with 30 people), we had developed a much more rigorous process and the goal of testing was to exercise all aspects of the code (clear box testing) to assure correct behavior and graceful failure if something went wrong. We built stubs and drivers (and sometimes simulated or modified operating system components) to generate every error condition that we could think of. Much of our code was operating system level (real time process control, low level I/O), at that time I was convinced that basis path testing (testing all possible paths through individual modules and integrated subsystems) was the only "true" path to testing. Five years later, I was working on systems that did not lend themselves to basis path testing - too much code was being run from libraries, we were no longer running stand alone systems, and frankly I was a little overwhelmed. The fallback plan at that point was to both assure that we got the intended functionality, and also to try and identify tests that broke the systems. The difference was that the box wasn't clear anymore... sometimes white, sometimes black. This was about the time I met Cem Kaner and James Bach, who were taking testing very seriously. This is also when I made the transition away from hands on development toward project management.

DDJ: What is the most interesting bug you have seen?
PH: The reason I chose to pursue computer science rather than electrical engineering, was that I found the real world messy. I liked the mathematical purity of software, assuming that it would either work or not work as a direct result of my skill and thoroughness as a developer or designer. Consequently, the most interesting (and humbling) bugs to me were probably bugs that I found in operating systems and compilers which I assumed would behave consistently and correctly. Foundation component bugs are particularly nasty, because once you find one the rookies on the team begin assuming that ANY bug they encounter is probably in the operating system or compiler or comm routine and it is difficult as a team leader to help people sustain the appropriate levels of skepticism.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
PH: I would give testers the same guidance that I give analysts and project managers. I think the most important thing is to learn to make a clear distinction between what you KNOW and what you THINK you know. The ability to recognize and capture assumptions before you mistakenly move them into the "truth" category is an essential skill. Recognizing and documenting assumptions is a critical skill because there is rarely time to exhaustively explore any question. Assumptions let us skip areas we believe are safe as we progress on our inquiry, but capturing them as assumptions reminds us that we may need to go back and revisit them systematically if we hit a dead end. You know someone is having an assumption problem when something happens and you hear them say "This is impossible! This CAN'T be happening!". Denying reality suggests that your mental model includes some assumptions that are incorrect.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PH: I don't imagine this will be an original answer, but I think that the exploding complexity of the systems that we must build and test and manage is going to be the greatest challenge we face. I've been building software since 1975. I've been doing it for money since the early 80's. Systems we thought were complex in 1980 consisted of a million machine instructions. Today, systems consist of dozens or hundreds of components of comparable complexity. Some are part of the application, some are part of the runtime environment, some are part of the foundation (OS, comm layer, DB layer, security infrastructure). I don't think that we have dramatically improved software quality much during the last 25 years. I think Cem Kaner is the one who pointed out to me that our skill designing and testing systems is rising linearly, while the size and complexity of the systems we build is rising exponentially. This is a daunting realization that makes me happy that I'm concerned with coaching and training project managers these days rather than taking responsibility for assuring system quality in modern systems. The scariest bit of technical graffiti that I ever saw said: "COMPLEXITY DECAYS". Without a major improvement in the way we conceptualize and conduct system design, implementation, and testing - I must imagine that the whole thing is going to collapse under its own weight sometime soon.

DDJ: Is there anything else you would like to say?
PH: Perhaps in light of the previous answer, I would like to say, "don't give up". I think that the challenges ahead are daunting, but we need to think of new and better ways to build and test systems so that the collapse that appears inevitable can be avoided. From a design perspective, this means partitioning systems into components that are well defined and defend themselves against errors and contamination from adjacent and dependent systems. We must get better about designing systems for testability and ease of diagnosis. A surprising number of organizations today are designing and testing in much the same way that they were 25 years ago. These processes MUST be replaced by more thoughtful approaches. It is easy for organizations to think that what worked last year will work this year and next year. The constant increase in complexity that we witness in every release of everything suggests that simplified thinking (test a few cases to make sure the new version still works) is doomed. I think the leaders in the field get it. The average practitioner in the field is falling further behind each year. We need to reach out to the average and below average performers and help them improve their practices if we must interact with their systems. Don't give up on your efforts to improve practice just because it is a daunting task.

I believe there is a mistaken belief that the art of performance testing and capacity planning is not relevant because of the ever increasing capacity of our hardware and networks. The complexity of our systems make performance testing extremely challenging, but identifying and resolving sources of performance grief is something that is understood by few and always in demand.

Posted by The Braidy Tester at 07:30 AM  Permalink