THE BOOK OF TESTING

Thoughts From a Braidy Tester

April 2007

You are not done testing unless…you have given your application's configurability options a thorough going-over. Applications can be configured via many different avenues: global and per-user configuration files and global and per-user registry settings, for example. Users tend to appreciate being able to customize an application to look and work exactly the ways they want it to look and work; they tend to get grumpy if all those customizations are nowhere to be seen the next time they launch the application. Interoperability with other instances of the application and with other applications fall into this same boat: most users expect a certain level of interoperability between applications, and they often get grumpy if your application does not meet that bar. Window interactions show up here too. Here are a few items to kick off your configuration and interoperability brain storming session:

Configuration

• Verify settings which should modify behavior do


• Verify settings are or are not available for administrators to set via policies, as appropriate


• Verify user-specific settings roam with the user


• Verify registry keys are set correctly, and that no other registry keys are modified


• Verify user-specific configuration settings are not written to machine or global registry settings or configuration files


• Brainstorm how backward compatibility problems might occur because a setting has moved or changed and thus broken functionality in a previous version or changed default values from a previous version

Interoperability

• Verify clipboard cut, copy, and paste operations within your application


• Verify clipboard cut, copy, and paste operations between your and other applications


• Verify drag and drop operations within your application


• Verify drag and drop operations between your and other applications

Window Interactions

• Verify z-order works correctly, especially with respect to Always On Top windows (e.g., online help)


• Verify all modal dialog boxes block access to the rest of the application, and all modeless dialog boxes do not


• Verify window and dialog box focus is correct in all scenarios


• Verify window size is correct after restore-from-minimize and restore-from-maximize


• Verify window size is correct on first launch


• Verify window size is correct on subsequent launches


• Verify window size is maintained after a manual resize


• Verify multiple window (i.e., MDI) scenarios work correctly


• Verify window arrangement commands (e.g., Cascade All) work correctly


• Verify multiple instances of the application work correctly across all scenarios (e.g., that a modal dialog box in one instance of the application does not disable interactivity in another instance)

Posted by The Braidy Tester at 07:30 AM  Permalink |

Bliss is an interesting guy, and not just because he has only one name! I encountered Bliss when I attended his turned-out-to-win-the-people's-choice-award talk at STAR East last year. Although currently Engineering Manager at Captaris, Bliss has also spent time spelunking Kentucky's caves for the Center for Cave and Karst Studies. He focuses the other direction too, working with Minor Planet Research, Inc. and the Lowell Observatory to look for and observe Big Chunks Of Rock which have a chance of running into our local Big Chunk Of Planet.

Like I said, Bliss is an interesting guy! Here's what he has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
Bliss: I started at a small software company that did not have a quality department. All testing was handled by technical support engineers between calls. As new code was being readied for release, the tech support department would begin testing it to validate its effectiveness. I began complaining about not having a structured test organization and feared that we were going to have a catastrophic failure at some point. The squeaky wheel gets the grease, as it were, so they charged me with putting together a Quality Assurance group.

It was clear from this that this organization did not initially take testing seriously, so I had to work hard to raise awareness of the value this group added to the software lifecycle. The importance of reporting and communication became evident and I found myself, as solitary tester and manager, spending a great deal of time putting together testing reports. It wasn’t just about the testing – it was about communicating what value the testing was giving back. I wasn’t simply finding bugs; I was reporting the areas where we could be more confident in the software and pointing out areas of higher risk.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
Bliss: That would have to be how misunderstood the testing process is by non-testers. All too often testing is perceived by non-testers as being the magic at the end of the cycle that magically puts the “quality” into the product. Many organizations leave all the quality planning for the end of the development cycle, pushing all responsibility onto the QA group. Of course, we all know that quality beings with the design, continues with the development, and is only checked by QA. Designs can be made with quality in mind and software can be coded with quality in mind, but, try as you might, you cannot test quality into the product.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
Bliss: Testers to know and do: Plan testing to find every single bug – spend every working moment improving the test planning, and don’t let any bugs get through. N.B.: Bugs happen. Bugs are pretty sneaky. Bugs get through your best efforts. There is no way around that – you are not going to find every one. It feels awful when something you think should’ve been caught, wasn’t. Don’t get bogged down in the “shoulda, woulda, coulda” thinking. It doesn’t help. When something gets through, focus on how it got through and change your plan for next time. Learn from it.

Developers to know and do: Write perfect code. You must test your code yourself, on clean systems (not development environments), before giving it to anyone else. Use manual testing, use unit testing, use whatever you want, just give it a few different run-throughs before you call it, “good”. Then don’t be surprised (or offended) when a tester reports problems. Bugs get through your best efforts. The tester is not trying to make a statement about your work; they are trying to help improve your customer’s experience. Be positive. Work together. Make better products as a team.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
Bliss: Yes. Detailed, step-by-step test plans – they are about as useful as a record-and-playback automation script. It works the first time and has to be painstakingly modified with each release. The better approach is to have test plans that describe the purpose/goals of the test and the areas of the software to concentrate on. Let the experienced tester work out the details as they test. They are better at responding to the software’s behavior when they aren’t constrained by a step-by-step procedure (of course, when they find a defect, they will have to detail the steps to reproduce it in the defect report). Constraining their creativity will not help to find defects, and letting them enter unexpected data at unexpected times might surprise everyone.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
Bliss: To keep up with the agile trends in our industry. For too many years we’ve relied on volumes of step-by-step test scripts and documentation. It is easy to inundate the project stakeholders with meaningful-looking stacks of paper, showing our testing activities. Agile techniques move us away from the stacks of paper and force us to provide status and activity reports in other ways. Even defect trend reports might become less meaningful as the developers begin to use their own defect prevention techniques --with our assistance. QA organizations will find fewer defects in the code, but will continue to add defect-prevention value to the projects. Under those circumstances, finding a way to show the benefit of the QA organization to the stakeholders will be challenging.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have thought hard about security testing and made explicit decisions about which testing to do and to not do. Back in the day, when even corporate computers were unlikely to be connected to a network, security testing didn't seem that big of a deal. After all, even if a computer did get infected by a virus it couldn't do much damage! Nowadays, viruses and worms take down corporations' mail systems, even my mother is inundated by spam, and hordes of unknowing consumers host trojan applications doing who knows what under the service of who knows whom. Security testing is now officially a Big Deal. Here are but a few of the plethora of test cases to consider; for more details consult the many big thick security tomes for sale at your favorite bookseller.

• Pore through your source code, APIs, and user interface looking for potential
  • Buffer overrun attacks
  • Denial of service attacks
  • SQL injection attacks
  • Virus attacks
  • User privacy violations (e.g., including user identifying data in saved files)
• On Microsoft Windows OSs, use Application Verifier to ensure no NULL DACLS are created or used - and to check for many other potential security issues
• Verify security for links and macros is sufficient and works correctly
• Verify relative filenames (e.g., "..\..\file") are handled correctly
• Verify temporary files are created in appropriate locations and have appropriate permissions
• Verify your application functions correctly under different user rights and roles
• Verify your application functions correctly under partial trust scenarios
• Verify every input is bounds-checked
• Verify known attack vectors are disabled

Posted by The Braidy Tester at 07:30 AM  Permalink |

David Treadwell joined Microsoft two months after I graduated from high school. I don't know whether that makes him really old or me really young. <g/> Back in the day he wrote file server and TCP/IP code for Windows NT. Perhaps his most visible contribution to date, however, has been running the .Net Developer Platform team, otherwise known as the people who invented and developed Microsoft's .Net Framework. While there he shepherded .Net 1.0's creation and release and helped .Net 1.1 get out the door. About a year ago he moved over to the Windows Live Core team, where he is working on all sorts of nifty secret things which we will hopefully learn about before too long. Here is what David has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
DT: I’m going to take the liberty to give a two-part answer.

My first formal introduction to testing was a course I took in college. I was an electrical engineering major, and one of the senior-year courses was about silicon chip testing. Much of the course was about things like error-correcting codes and the like, which are not especially relevant to software testing. However, one principle from the course really stood out to me: the value of designing for testability. The course strongly made the point that one must consider testing early in the design phase of an engineering project because the cost of testing can be dramatically reduced if the design is built for testability.

The other major early exposure to test for me was in 1992 when I was developing the NT Winsock implementation. I had actually been at Microsoft for several years working on the SMB server, and while I had a fair amount of interaction with test, the relationship was somewhat distant. What changed when I was working on Winsock is that I was pretty much the sole developer responsible for the project, and the company had assigned exactly one tester to work on the project with me: Jim Gilroy.

At first, the way things worked is that Jim would write some code to invoke the Winsock APIs that I had written, then he would file bugs if he thought he found problems. I would typically find these bug reports annoying for a variety of reasons: sometimes they were outright bogus, sometimes they were of questionable legitimacy, and sometimes they were real and pointed out my own failings. Whatever it was, I usually felt like addressing these bugs was a distraction from the “real” work of writing new code.

Before too long, however, my code started getting out into the wild via early developer beta releases of NT. Real external people were using my code to write real apps. Predictably, these apps sometimes ran into real problems in my code. These problems were always VERY expensive to address, since I often did not have the app code or easy reproducibility of the issue. It was also very annoying to the people trying to write apps on top of this very new infrastructure.

Eventually, something that now seems so obvious just clicked for me: Jim was not an adversary. He was a key partner to me. If Jim’s test harness found problems in my code it was VASTLY easier for me to address the issue than if a customer ran into the problem. I came to appreciate just how selfishly valuable this resource was to me, and it changed my whole perspective on the test relationship. I listened much more to the suggestions that Jim made. I proactively gave him suggestions on things he should include in his tests. I spent a lot more energy looking into things before assuming it was a test bug. Through it, Jim was patient with me and the process, always focused on making the product better.

Jim’s tests eventually became the de facto standard for the manner in which a Winsock implementation should behave, and this was a small but very important part of the growth of the Internet. Without Winsock as a common platform on which to write applications like Mozilla, it would have taken much longer for the Internet to change society in the way it has.

My lesson from all of this: test is a key ally in building a great product. Treat test as a partner, not an adversary.

DDJ: What is the most interesting bug you have seen?
DT: Back in 1991, when working on the NT SMB (file) server, we were starting to do “dogfooding” of our system. We stored the NT source code on systems running super early versions of NT—essentially daily builds. These would crash on occasion, which was annoying to folks who wanted to synch or check in code. However, what was really bad was when there was corruption of the source files. When someone broke the build, there was appropriately a lot of negative feedback given to the person. However, when the person had checked in good code, but the server had “munged” it, people got outright pissed.

We had one data corruption bug in these dogfood source servers that was rare but persistent. Eventually, my boss asked me to drop everything until I got this bug fixed. Because it was uncertain whether the bug was in the server code or somewhere else in the system, management also assigned another developer from the core OS team, Mike Glass, to work on the issue with me.

Mike and I spent several weeks trying to instrument the system to track down the problem. Because the problem rarely reproduced, and because we only learned of corruption well after the bug occurred, it was a total nightmare to track it down. It was bad enough that I had dreams about memory patterns and assembly traces. Eventually, Mike tracked down the bug to a problem in the SCSI hardware that we had been using on those systems. Pascal Zachary mentions this story in his book Showstopper.

Lessons: do as much instrumentation as possible, tools matter, and be open-minded and persistent about where the problem may lie.

DDJ: How would you describe your testing philosophy?
DT: I largely described it in the story above about Jim Gilroy: test is an essential part of the software engineering process. Good, efficient test goes a very long way toward making a product great.

A corollary to this philosophy is that automated testing is much more powerful than manual testing. When things are automated, it is straightforward to put new code or changes “through the wringer” to ensure that it still works—or to find issues in it.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
DT: For the tester: be patient with your developers who might not yet see your value. If they are any good, they will come around eventually. Also, be tenacious about being the quality gate for the product. Do not expect perfection in the product, but assertively demand excellence. Think hard about the real scenarios that users will put the product through, and gauge the importance of your bugs based on that. Not everything is a P1, but when you find something that will truly be a real negative impact on users of the product, hold the team’s feet to the fire on getting it addressed.

For the developer: those testers may seem annoying, but understand that, in the long run, they are some of the best partners you have. You should MUCH prefer that they find the bugs instead of your customers.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
DT: The growing importance in the industry of services will bring some interesting challenges and opportunities to the test discipline. The challenges come from the fact that services have very different processes and constraints than typical packaged software: essentially continuous release cycles, 24x7x365 operation of the service, massive scale data centers. The opportunities come from the fact that services have some nice agility advantages over packaged software, and these advantages should allow more efficient testing in many dimensions.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you understand the performance characteristics of your application and the manner in which your product deforms under stress. Performance testing can seem straightforward: verify the times required to complete typical user scenarios are acceptable - what's so hard about that? Simulating those scenarios sufficiently realistically can be difficult, however. Even more so when it comes to stress testing! For example, say you are testing a web site which you expect to become wildly popular. How do you simulate millions of users hitting your site simultaneously?

I do not have any easy answers for these questions. What I do have are several scenarios and conditions to consider:

Performance

• Verify performance tests exist for each performance scenario, and are being executed on a sufficiently regular basis


• Verify performance targets exist for each performance scenario, and are being met


• Verify the performance tests are targeting the correct scenarios and data points


• Verify performance optimizations have the intended effect


• Verify performance with and without various options enabled, such as Clear Type and menu animations, as appropriate


• Compare performance to previous versions


• Compare performance to similar applications

Stress

• Run under low memory conditions


• Run under low disk space conditions


• Run under out-of-memory caused via automation (e.g., a use-up-all-available-memory utility)


• Run under out-of-memory caused by real world scenarios (e.g., running multiple other applications each having multiple documents open)


• Run under a heavy user load


• Run over a network which frequently drops out


• Run over a network with a large amount of traffic


• Run over a network with low bandwidth


• Run on a minimum requirements machine


• Open, save, and execute (as appropriate) from floppies and other removable disks

As you do all of this performance and stress testing, also check for memory and other resource leaks.

Posted by The Braidy Tester at 07:30 AM  Permalink |

I first met Michael Bolton several years ago when he and I both happened to be in San Jose and he invited me to a meeting of the Weinborg (i.e., students of Jerry Weinberg). Since then he has become something of a mentor to me as I attempt to wrap my head around this Exploratory Testing thing he is always running on about. Michael is a frequent speaker at testing and software development conferences (including, natch, Software Development West), teaches Rapid Software Testing all over the world (my review here), and is an active participant on every testing-related discussion list I know. He has that seemingly rare knack of asking questions which make you think without making you feel stupid or put down - which I appreciate as I find myself pondering his questions a lot! Here is what Michael has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
MB: Wow, that's a question with a lot of potential answers. To start with, I was brought up in a family where asking questions and seeking answers was encouraged. In that sense, I like to think that I've been testing all my life. When I started programming in the late 80s, I couldn't really separate the act of running my code and testing it; every time I believed something worked, there was a little daemon in the back of my head asking me "How do you know it works?" When I started working at Quarterdeck in the early 90s, I was in technical support. Our products were designed to provide memory management and multitasking for regular DOS applications, and there were zillions of those. Figuring out and reporting on why our products sometimes failed to achieve the expected result for our customers was very much a testing activity. In addition to that, everyone in technical support and most of the other people throughout the company tested our products as soon as the developers released builds.

My first formal job called "testing" was at Quarterdeck, co-ordinating testers and test activities for DESQview/X, QEMM, and DESQview. In that role I did a good deal of testing myself, partly to find problems on my own, and partly to reproduce and confirm reports that were coming into the development department. At some point during that period, I was given a copy of Cem Kaner's Testing Computer Software to read, and I devoured it; it was the best piece of technical writing I had ever seen. Later I became a program manager. The way I approached the job, it exemplified James Bach's definition of testing--"questioning a product in order to evaluate it". In fact, I found that it wasn't enough to question the product; I also had to question all of the systems around the product--the development team, the installation team, build and release control, technical support, marketing, and so forth. In those areas, the great performers all had that questioning attitude too, and the result was some pretty amazing software for its time.

In each of these roles, I developed a greater appreciation for the richness and complexity of testing. The unifying theme is the process of trying to make sense of things by asking questions and seeking answers.

DDJ: How would you describe your testing philosophy?
MB: My testing philosophy is closely aligned with Cem Kaner and James Bach, and it has been profoundly influenced by them, and by Jerry Weinberg. It's centered around the idea that testing is a process of investigation, finding out how stuff works in an open-ended search for information. That leaves a lot of room for testers both to generalize and to specialize. It suggests that testers need not, should not, all have the same skills. It means that all testers can bring new perspective and insight based on their own unique knowledge and experience. It also means that testers should remember to interact with the application as its intended user does. It's fun--and it might even be important--to do lots of clever automation stuff, to make plans, to draw diagrams, to write scripts and so on--but those things aren't the testing effort--they're tools that serve the testing effort. Over the last few years, the focus has drifted from "test everything" (which is impossible anyway) to "automate everything" (which is not only is impossible, but which may not serve the testing effort particularly well). I worry a lot about that; it feels to me like replacing one silver bullet with another.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
MB: I like the fact that you've divided the question into parts, because I think the answers are quite distinct. To me, the most important thing is that there are plenty of important things, and it might not be such a great an idea to settle on only one.

My answer today for testers: the most important thing is that testing has a lot of facets, but it's all being done for people--lots of people. We're providing information to management about the state of the product; we're assisting the developers in identifying risks and problems that could reduce the value of their work; we're trying to anticipate the problems that users will encounter in the product. Those different missions, for those different groups, mean that we can use lots of different approaches to find important problems. It's important to work with our managers and our project sponsors to negotiate our missions so that we can exploit and develop our skills in the ways that best serve the project.

Cem has recently been giving presentations in which he remarks that testing can't be hard science any more. In the old days, everybody who worked on the program could understand the whole thing--there were only a few thousand lines of code, a tractable number of variables. And to a great degree, product scope was limited enough that proving a program mathematically correct might have been possible and even useful. Systems today are too complex for that stuff. We can't hope to comprehend programs that have hundreds of thousands or millions of lines of code, never mind the interactions between them. We can't be sure of the reliability of the underlying libraries and operating systems, either. There are so many contexts in which a piece of software can be used that we have to settle for something more like a social science model, where there are enormous numbers of free variables, where answers aren't definitive, and where subjectivity of the analysis is a given. In that kind of realm, we have to accept the fact that we can't get certainty, since a changing context can change everything we think we know about the product. On the other hand, we can still do excellent work when we give people information that they value. Cem referred to the social sciences as giving us "partial answers that might be useful"--and I think that's bang-on. To me, testing really is like that, and I think it's important for testers to embrace the idea--and to help our colleagues in other roles to understand it, too.

For developers, there are a few things that are worth knowing about testing. The first is that we testers are trying hard to make you--and the rest of the project team--look good. If we sometimes seem a bit negative, it's because we're often focused on risk, where developer-folk are focused on the positive outcome. Those differences in mindset are natural and valuable, and both are necessary if we want to know how the product can work and how it might fail. The second point is that developers can really help us testers by building products that are more testable--by which we mostly mean controllability and visibility. It helps to speed testing enormously if we can throw some automated tests at a scriptable interface, and if we can obtain a detailed log of stuff that the application is doing. A third point for developers is that the more bugs we testers find, the worse our test coverage gets. It takes some amount of time to perform a test, but it takes a good deal longer to investigate and report a bug. That leads to a paradox: the more bugs we find, the less time we have to find bugs, because the time that we spend on investigation and reporting is time that we can't spend on the design and execution of new tests. That leads to the fourth thing, which is that unit testing--typically in a test- or feature- or behaviour-driven development model--can really help to get some of the trivial stuff out of the way. To the extent that I'm doing TDD myself, I'm finding that it really helps to make my code more robust while improving my programming speed. Jerry Weinberg told me once that in the old days, programmers needed to write lots of unit tests because the machinery was so unreliable; now that the machinery is really robust, the operating environments and programming languages and runtime libraries are hideously and complex, so there's a lot of risk there. Unit tests address that risk really early in the process, which keeps feedback loops nice and tight. In addition to that, they're a super way to do a great deal of regression testing--right at the source level.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?
MB: Oh, there's lots of stuff--"mythodology"--that I think we should be able to put to bed for a long, long nap. The first is the idea that we can use words like "always" or "never", or any other absolute language with reference to testing. One of the critical thinking games that James Bach and I like to play is called "Don't, Because, Instead". The object of the game is to take some practice that sounds like a good thing, give a reason why one shouldn't use it, and offer a new practice to use instead. The game works best when there's an even number of players and an odd number of practices, such that after a few rounds, you make a powerful argument in favour of the practice that you argued against in an earlier round. The game is designed to emphasize the potential value of each practice, and the foolishness of any claim that there's such a thing as a universal best practice. Context-driven testers try hard not to believe that--even to the extent that we recognize there are contexts in which context-driven thinking itself might not be such a hot idea.

Another thing that I would like to see go away is the obsession that some of us seem to have with numerical measurements of things to do with testing, especially with evaluation of the test effort with estimation. I'd prefer to see a focus on qualitative (and qualified) assesments instead. A lot of people who speak at conferences and write in magazines and books present these really elaborate mathematical models, but when you look at the way the numbers are calculated, there are so many assumptions, so many free variables, so much bogus precision that can be undermined by a tiny change in circumstances. Jon Bach and James are focusing on the notion of the testing story, and how it aligns with the product story, as being much more important. Numbers without a story attached are abstractions by definition, data without information.

When the project sponsors ask you for an estimate in how long it will take to test something, I think they're asking a question that doesn't have a good answer. There's an infinite number of tests that you could run, so you could take forever. That won't fly with the managers. You don't want to stop testing before the product ships, either. So here's one way to estimate the testing effort on a project: In my experience, the project owners have a ship date in mind anyway, so tell them that you'll test the bejeebers out of the product until they want to release it--and that you'll provide them with all the information that you possibly can until then. This approach isn't precise, but it's 100% accurate--and it's quick.

I think the current certification fad can and should be ignored until the certifiers actually watch you testing something and put some effort into examining your thought processes. Bugs don't follow bodies of knowledge, and they don't fill out multiple choice tests; they're often far subtler than that. The testing mindset has to be at least as subtle. Finding bugs, reporting information clearly, developing testing strategies, adapting to changing situations, understanding that nomenclature can vary from one context to another--those are only a few things that a tester has to be able to do well. A certification process that doesn't test for that isn't really certifying testers; it's certifying clerks. One of the people from one of the certifying organizations said to me once that he felt his company's certification should be viewed more as a learner's permit than as a real evaluation of expertise. I suddenly developed a lot more respect for him; I just wish that his company's marketing materials were as forthright as he was.

DDJ: Anything else I should ask you about?
MB: Ah--that's a question that I think you learned to ask from Jerry Weinberg. You could ask me about where testers might start a serious, advanced study on testing and on software quality--and my suggestion is to start by reading Jerry's stuff (his "author name" is Gerald M. Weinberg), joining the SHAPE Forum, and attending the AYE Conference. You could also ask how to get in touch with me so that I can answer more questions--an offer that our community tries to extend to everyone. My Web site is at http://www.developsense.com, and I love talking about testing in pretty much any available medium. Please feel free to drop me a line any time!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have vetted your application for Year 2000 issues. Even though we are now well past that date, Y2K issues can crop up with the least provocation. Those of you on some form of Unix (e.g., all you Apple users) have another Y2K-ish situation coming up in 2038 when that platform's 32-bit time data structure rolls over. Oh, and as long as you're looking at date-related functionality, you may as well look for other date-related defects as well, such as general leap year handling.

• Verify dates entered with a two digit year from 1 Jan 00 through 31 Dec 29 are interpreted as 1 Jan 2000 through 31 Dec 2029
• Verify dates entered with a two digit year from 1 Jan 30 through 31 Dec 99 are interpreted as 1 Jan 1930 through 31 Dec 1999
• Very dates at least through 2035 are supported
• Verify dates in leap years are correctly interpreted:
  • 29 Feb 1900 should fail
  • 29 Feb 1996 should work
  • 29 Feb 2000 should work
  • 31 Dec 2000 should work and be identified as day 366
  • 29 Feb 2001 should fail
• Verify other interesting dates are correctly interpreted and represented, including:
  • 31 Dec 1999 should work
  • 1 Jan 2000 should be unambiguously represented
  • 10 Jan 2000 (first seven digit date)
  • 10 Oct 2000 (first eight digit date)
• Verify entering "13" for the month in year 2000 fails

Posted by The Braidy Tester at 07:30 AM  Permalink |

S. Somasegar runs the Developer Division at Microsoft. That means he's my boss. (If you ignore the three levels of management in between....) It also means he's ultimately responsible for everything you love and hate about Microsoft's Visual Studio and Expression tools. Evidently that still leaves him some free time, because Microsoft's India Development Center in Hyderabad is under his wing as well. And his eponymous blog is proving to be the most reliable (i.e., he's actually still posting on a regular basis!) Microsoft Big Wig blog around.

One of the things I like best about Soma is that he used to be a tester. Here's what Soma has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
Soma: My first introduction to testing was when I joined the OS/2 team as a Software Design Engineer in Test owning some of the kernel components of the operating system. There were two main reasons why I was excited about testing software – 1) I knew that part of a tester’s job was to represent the customer, understand what the customer would want to do with the product and then apply that knowledge and intuition to testing to ensure that the product lived up to the customer’s expectations on functionality and quality 2) The act of debugging a software problem is exhilarating and literally addictive.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
Soma: Testing is quite fun and challenging if you are interested in trying to break things apart, have fun in debugging things and problem solving. You get to learn the breadth of the system (software) and understand how software components work together to enable scenarios. From a functional management perspective, test management is probably the most complicated one.

DDJ: How would you describe your testing philosophy?
Soma: The tester has a unique responsibility to put himself/herself in the customer shoes and ensure that the product is customer ready by the time we ship – the scenarios stick together, the quality, performance and usability are up to standard and that the customer has a great experience when they use the product.

DDJ: What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
Soma: The most important thing for a tester to know is that they are the 1st customer for the software that is being developed and they absolutely represent the customer view when it comes to testing and quality. They have to be involved hand in hand with the developer as the software is being developed and not as an after-thought. Developers need to keep in mind software testing right from the beginning and ensuring that testability hooks are built in. Developers are as much responsible for the quality of the software and the experience as testers are.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
Soma: Software development continues to be constrained by testing, at least at Microsoft. We often talk about the biggest constraint to delivering software is our test bandwidth. This really means that we need to do a much better job of involving test right from the design phase, ensure that we apply the right resources, bandwidth and IQ to truly make breakthrough progress in terms of tools related to test automation, test prioritization, code coverage and the like. We should also think about what the roles and responsibilities are for a developer in terms of ensuring quality. There are teams at Microsoft now that are experimenting with different models where an engineering team is responsible both for development and testing and not really differentiating between a tester role and a developer role. It will be interesting to see which of these models evolve as we take software engineering (development and test) to the next level.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you understand how your application handles being installed over previous versions of your application, and having the operating system upgraded out from under it. You may want to test installing a previous version over, or side-by-side to, the current version as well. Consider whether to cover all three combinations: upgrading just your application, upgrading just your operating system, and upgrading both the operating system and your application.

Application Upgrade

• Verify upgrading over a previous version replaces appropriate files and no others


• Verify installing this version side-by-side to previous versions works correctly


• Verify the correct files do and do not exist after an upgrade, and that their versions are also correct


• Verify default settings are correct


• Verify previously existing settings and files are maintained or modified, as appropriate


• Verify all functionality works correctly when the previous version(s) and/or the new version is set to Run From Network


• Verify any features and applications dependent on files or functionality affected by the upgrade work correctly

Operating System Upgrade

• Verify upgrading over a previous version replaces appropriate files and no others


• Verify all functionality works correctly


• Verify any features and applications dependent on operating system files or functionality affected by the upgrade work correctly

Posted by The Braidy Tester at 07:30 AM  Permalink |