THE BOOK OF TESTING

Thoughts From a Braidy Tester

February 2007

Pradeep Soundararajan is one of the shining and rising stars in the Indian testing community. I find Pradeep's blog Tester Tested a fascinating read; while his viewpoint is in many ways quite similar to mine it is also completely different. His passion for testing, learning about testing, and learning about himself seems boundless. He currently represents Satisfice, Inc., testing, consulting, and coaching; he is also the founder of the Bangalore Skilled Testers Community.

Michael Bolton says "Pradeep's first language is not English--his first language appears to be testing." Here is what Pradeep has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
PS: The first time I heard and saw testing being done was in a movie "Edison - The Man" I watched in my teen age years. Something that caught my attention was Sir Thomas Alva Edison's design of experiments to test each of his inventions. It took me seven years from that time to become someone whom the industry addresses as a tester. It then took me two and half years to realize that a more sophisticated tester was growing within me, thanks to people like James Bach, Jerry Weinberg and Michael Bolton who introduced me to skilled testing. Since then I have been focusing on skills and each day I feel I am being introduced to something new in testing. Every day seems to be like a first introduction to testing!

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
PS: The different ways in which it makes me think.

DDJ: What is the most interesting bug you have seen?
PS: I was testing speech recognition program that runs on a mobile phone. On exploring the program, I became aware of a language pack that resides in the program. There's a different pack for each country. I do not have an American accent and so I was a bit hesitant to test it. However, I had to do it. The program was inconsistent in responding to my accent. In one of out every ten trials, I saw a response from the program and logged it as an issue. The developers (who were in another country) came back saying that the issue could have been my accent, because none of them was able to reproduce the problem. We had a teleconference and the developers recorded my voice samples and tried using those as test files to reproduce the issue that I logged. Meanwhile, I was forced to complete the test cycle and I was uncomfortable doing that. I came back to my cubicle, took the phone and let out my frustration by keeping the phone near my mouth and shouting "&$*^ &%$#".

Line 556: 30678 popped up on the screen. It took me a couple of seconds to realize the program had crashed. I tried reproducing it and each time it showed Line 556: 30678.

How could I report this crash? Would I need to mention that I said, "&$*^ &%$#"?

I tried investigating it and I finally found the reason. It wasn't the words that I mentioned that had crashed it. The problem was the audio gain. The program crashed when it received an input signal above a certain limit. I then showed a magic trick to my colleagues using the same phone and software. I told them, "My name is very powerful to produce a crash. I asked them to hold the hold the phone and to shout my name. "That's why PRADEEP is a GOOD TESTER!"

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
PS: On the Indian sub continent—maybe in other places too—a tester tends to stop testing after five years of experience; he or she gets promoted to Test Lead or Senior Tester, or to some other role like development or product management. Remaining as a tester itself is a big challenge in such places over the coming years.

I see what Michael Bolton suggested me to see. “Note that when testers migrate to other roles, they do less testing, but there's also a draining effect on the testing community; we lose experienced testers and replace them with less-experienced testers.”

DDJ: Is there anything else you would like to say?
PS: Oh yes! I want to say something about the importance of testers writing blogs. Blogs allow a novice tester to understand how an expert is thinking and to become inspired to write their own blogs. I would request all testers to blog the experiences that they feel could benefit the testing community. We are unfortunate that we don't have same experiences in life and it would be great if the community had a lot of stories and experience reports to read and learn from. Perhaps I came to know you through your blog Michael!

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing yet unless...you have verified how your application handles various network configurations and events. In times past you could more or less count on stability in the network - if the computer was connected to a network when your application started, it would almost certainly remain connected to that network while your application was open. Sure, some doofus might cut the main feed with a backhoe or yank the wrong cable in the router closet. The chances of something catastrophic happening were low enough, however, that bugs of the form "Disconnect your computer from the network while the application is opening a twenty megabyte file from a network share" tended to be Won't Fix'd with dispatch under the premise that "No user is going to do that".

Oh how times have changed! Users these days are more likely than not to be connected to a wireless network which drops out on a regular basis. Users who start out connected to a wired connection may undock their computer and thus disconnect from that network at any time. Net-over-cell phone is becoming ever more prevalent. Network-related "We'll fix that if we have time, maybe" issues are now problems which directly affect your customers on a regular basis. And so it is important to check the following:

• Connecting over a network which supports only IPv4
• Connecting over a network which supports only IPv6
• Connecting over a network which supports both IPv4 and IPv6
• Connecting over an 802.11a wireless network
• Connecting over an 802.11b wireless network
• Connecting over an 802.11g wireless network
• Connecting over an 802.11n wireless network
• Connecting over a GPRS (cell phone) network
• Connecting from a multi-homed machine (i.e., one which is connected to multiple networks)
• Connecting via a 28.8 modem
• Connecting via a 56k modem
• Connecting over a network other than the one inside your corporate firewall
• Connecting over a network which requires user authentication on first access
• Connecting over a network which requires user authentication on every access
• Passing through a software firewall
• Passing through a hardware firewall
• Passing through Network Address Translation
• Losing its connection to the network
• Losing its authority to connect to the network
• Joined to a workgroup
• joined to a domain
• Accessing documents from a network location which requires user authentication
• Performing a Print Preview to a network printer which is disconnected or otherwise unavailable

Posted by The Braidy Tester at 07:30 AM  Permalink |

Brian Marick has done it all: Programming. Testing. Managing. Testing consultant. Long about 2001 he dove into Agile, so much so that he was one of the authors of the Manifesto for Agile Software Development and has served as chair of the board of the Agile Alliance. These days he calls himself an Agile consultant and advocate. He hasn't forgotten everything he has learned about testing however, for those learnings shape his Agile work in many ways. Here is what Brian has to say:

DDJ: What was your first introduction to testing?
BM: It was 1981, just around the time TCP was introduced. I wanted to write networking software. My first boss said, "We really don't know how good you are, so we're going to put you in testing. At least there, you can't do any harm."

DDJ: What did that leave you thinking about the act and/or concept of testing?
BM: Well, first, that I needed to "graduate" from testing and get to my real work, programming. In those days, a lot of testers were either new hires on probation or programmers who hadn't made the grade.

That was the bad part. The good part was that I was assigned to test the code of a star programmer. I found 60-some bugs ("can't do any harm", indeed). That left me with the conviction that I could look like a much better programmer than I was, provided I tested my code well. That turned out to be true - except that today I wouldn't say I "looked like" a better programmer. I'd say I really *was* a better programmer.

DDJ: How would you describe your testing philosophy?
BM: Nowadays I focus on testing in Agile projects. They still need after-the-fact, product-evaluation testing, though I'd focus it more on testing "ilities" like performance, security, and usability.

I am no fan of having people repeat tests by following scripts. You might need a few of them, but the vast majority of repeated tests should be repeated by machine (that is, should be automated). If the tests are too expensive to automate, you most often (1) have programmers who don't care about testing, (2) have lousy relations between the testers and programmers, or (3) are automating wrongly. It's better to fix those problems than patch them over with scripted manual tests.

I'd want manual testing to be free-form and exploratory. And I'd want some exploratory testing no matter how good our automated test suite is. Writing automated tests puts you at a remove from the program: you're writing *about* using it, not using it. The act of putting hands on opens up a different kind of imagination -- we're tactile animals, after all -- and lets us notice and exploit oddities.

But Agile projects open up more opportunity for testers. Agile projects almost always reject requirements documents or rigorous specifications. Among their other problems, those documents are way too abstract. Abstraction is swell for identifying and concentrating on what you know or suspect is important, but they let you skim over what you think are irrelevant details.

*Examples* are good for revealing that irrelevant details are really relevant, or that two people who seem to be agreeing on the meaning of some abstract term are in fact disagreeing. For that reason, I have a sticker on my laptop's lid that says "an example would be handy right about now". In meetings that seem to be bogged down or skipping over something important, I hold up the laptop and repeat the slogan. Even better is when someone else points at the laptop and says it before I do.

A test is an example of how the product should work. So one of the tester's new jobs is to be part of product definition conversations and make sure there's agreement on concrete examples, not just on vague statements. And then make sure the product is exercised against those examples when it's built.

DDJ: What do you think is the most important thing for a tester to know?
BM: I don't know what's most important, but I've come full circle on the matter of automation. In the 80's, I thought it was ridiculous for tests not to be automated; therefore, all testers should know how to program. In the 90's, largely due to the influence of Cem Kaner, I came to realize the power of exploratory testing and how often automation was a waste of money in comparison. In that decade, I was accused at conferences of being "against test automation." That was a wild exaggeration, but I definitely erred on the side of manual testing.

Now, in the naughts, I'm back to favoring automation. That's because of three things. The first is the popularity of testing among Agile programmers, which means that they write far more testable programs. The effort to automate is much lower. The second is the availability of tester-friendly languages (my favorite being Ruby). Compared to the C of my day or the Java/C# of today, these languages make automation much easier. The third is the way Agile programmers depend on tests to guide and pace their programming. That means a test isn't useful once a month, when the programmers drop a new test build on the testers; it's useful from day one as the programmers run it over and over as a part of writing their code.

If you'll permit me some self-promotion, I'll mention here that my book Everyday Scripting with Ruby has just been published. Testers are the main audience.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
BM: For testers to influence the project the way they should, they have to be able to write tests that (1) can be understood by non-technical businesspeople so that they can be part of the project-long design discussion, (2) don't require programmers to write a lot of support code (that is, not more code than they write to support their own unit tests), and (3) are easy to edit, so that they can evolve as the product does.

The tools to support that are inadequate. Even if they were, we have a lot to learn about how satisfy the businesspeople, the programmers, and the changing program, all at the same time.

Posted by The Braidy Tester at 07:30 AM  Permalink |

You are not done testing unless...you have thoroughly tested your application's Open, Save, and Save As functionality. I don't know about you, but I get grumpy when my work disappears into thin air! For many applications, if data cannot be saved and later regurgitated with full fidelity, the application may as well not exist. Thus it is important to verify the correct thing happens under the following conditions:

• Open each supported file type and version and Save As each supported file type and version. Especially important is to open from and save as the previous version of your native format. Customers tend to get grumpy if upgrading to a new version means they can no longer open old documents! And they tend to not upgrade if they do not have a simple way to share documents created in the new version of your application with those poor souls still languishing on the old version.
• Open each supported file type and version and Save. If the file type and version can be selected during a Save operation (as opposed to a Save As operation), Save to each supported file type and version. More usually, Save saves to the current version only.
• Roundtrip from each supported version to the current version and back to the previous version. Open the resulting file in that version of your application. Does it open correctly? Are new features correctly converted to something the previous version understands?
• Open files saved in the current version of your application in previous versions of your application. If the document opens, how are features added in the new version handled? If the document does not open, is the resulting error message clear and understandable?
• Open from and Save and Save As to different file systems (e.g., FAT and NTFS) and protocols (e.g., local disk, UNC network share, http://). The operating system generally hides any differences between types of file systems; your application probably has different code paths for different protocols however.
• Open, Save, and Save As via the following mechanisms (as appropriate):
  • Menu item
  • Toolbar item
  • Hot key (e.g., Control+S for Save)
  • Most Recently Used list
  • Microsoft SharePoint document library
  • Context menu(s)
  • Drag-and-drop from the file system explorer
  • Drag-and-drop from your desktop
  • Drag-and-drop from another application
  • Command line
  • Double-click a shortcut on your desktop
  • Double-click a shortcut in an email or other document
• Open from and Save and Save As to the following locations:
  • Writable files
  • Read-only files
  • Files to which you do not have access (e.g., files whose security is set such that you cannot access them)
  • Writable folders
  • Read-only folders
  • Folders to which you do not have access
  • Floppy drive
  • Hard drive
  • Removable drive
  • USB drive
  • CD-ROM
  • CD-RW
  • DVD-ROM
  • DVD-RW
• Open from and Save and Save As to various types and speeds of network connections. Dial-up and even broadband has different characteristics than that blazing fast one hundred gigabyte network your office provides!
• Open files created on (and Save and Save As to as appropriate):
  • A different operating system
  • An OS using a different system locale
  • An OS using a different user locale
  • A different language version of your application
• Open from and Save and Save and Save As to filenames containing
  • The Text Entry Field YANDY list, as appropriate
  • The Filenames YANDY list, as appropriate
  • The Invalid Filenames YANDY list
  • Spaces
• Cause the following to occur during Open, Save, and Save As operations:
  • Drop all network connections
  • Fail over to a different network connection
  • Reboot the application
  • Reboot the machine
  • Sleep the machine
  • Hibernate the machine
• Put AutoSave through its paces. What happens when you AutoSave every zero minutes? Every minute? With a very big document? If the AutoSave timer is per document, what happens when multiple AutoSaves kick off simultaneously, or while another AutoSave is in progress? Does file recovery from AutoSave work as you expect? What happens if the application crashes during an AutoSave? During recovery of an AutoSaved document?
• Save and Save as in the following conditions:
  • No documents are dirty
  • One document is dirty
  • Multiple documents are dirty and the user chooses to save all of them
  • Multiple documents are dirty and the user chooses to save none of them
  • Multiple documents are dirty and the user chooses to save only some of them

Posted by The Braidy Tester at 09:30 AM  Permalink |