You Are Not Done Yet: Setup

You are not done testing yet unless...you have tested your program's setup process under the following conditions. Some of these terms are specific to Microsoft Windows, but other operating systems generally have similar concepts.

• Installing from a CD-ROM/DVD-ROM
• Installing from a network share
• Installing from a local hard drive
• Installing to a network share
• Installing from an advertised install, where icons and other launch points for the application are created (i.e., the app is "advertised" to the user), but the application isn't actually installed until the first time the user launches the program. Also known as "install on demand" or "install on first use".
• Unattended installs (so-called because no user intervention is required to e.g., answer message boxes), aka command line installs. This can become quite complicated, as the OS's installation mechanism supports multiple command-line options, and your application may support yet more. Oh the combinatorial testing fun!
• Mass installs, via an enterprise deployment process such as Microsoft Systems Management Server.
• Upgrading from previous versions. This can also become quite complicated depending on how many versions of your app you have shipped and from which of those you support upgrades. If all of your customers always upgrade right away, then you're in good shape. But if you have customers on five or six previous versions, plus various service packs and hotfixes, you have a chore ahead of you!
• Uninstall. Be sure that not only are all application-specific and shared files removed, but that registry and other configuration changes are undone as well. Verify components which are shared with other applications are/not uninstalled depending whether any of the sharing apps are still installed. Try out-of-order uninstalls: install app A and then app B, then uninstall app A and then uninstall app B.
• Reinstall after uninstalling the new and previous versions of your application
• Installing on all supported operating systems and SKUs. For Microsoft Windows applications, this may mean going as far back as Windows 95; for Linux apps, consider which distros you will be supporting.
• Minimum, Typical, Full, and Custom installs. Verify that each installs the correct files, enables the correct functionality, and sets the correct registry and configuration settings. Also try upgrading/downgrading between these types - from a minimum to complete install, for example, or remove a feature - and verify that the correct files etc are un/installed and functionality is correctly dis/enabled.
• Install Locally, Run From Network, Install On First Use, and Not Available installs. Depending on how the setup was created, a custom install may allow the individual components to be installed locally, or to be run from a shared network location, or to be installed on demand, or to not be installed at all (I use that one a lot for clip art). Verify that each component supports the correct install types - your application's core probably shouldn't support Not Available, for example. Mix-and-match install types - if you install one component locally, run another from the network, and set a third to Install on First Use, does everything work correctly?
• Install On First Use installs. Check whether components are installed when they need to be (and not before), and that they are installed to the correct location (what happens if the destination folder has been deleted?), and that they get registered correctly.
• Run From Network installs. Check whether your app actually runs - some apps won't, especially if the network share is read-only. What happens if the network is unavailable when you try to launch your app? What happens if the network goes down while the application is running?
• Verify that "normal" or limited-access (i.e., non-admin) users can run the application when it was installed by an administrator. Especially likely to be troublesome here are Install On First Use scenarios.
• Verify the application works correctly under remoted (e.g., Microsoft Remote Desktop or Terminal Server), and virtual (e.g., Microsoft Virtual PC and Virtual Server) scenarios. Graphics apps tend to struggle in these cases; I've seen apps just plain refuse to boot when running under Terminal Server.

Update 25 Aug 2006: Reader Jon B. suggests some additions to my list:

• Perform a Typical install followed by a Modify operation to add additional features.


• Perform a Custom install followed by a Modify operation to remove features.


• Perform a Typical install, delete one or more of the installed files, then perform a Repair operation.


• Perform a Custom installation that includes non-Typical features, delete one or more of the installed files, then perform a Repair operation.


• Patch previous versions. Patching is different from an upgrade in that an upgrade typically replaces all of the application's installed files, whereas a patch usually overwrites only a few files.


• Perform a Minor Upgrade on a previously patched version.


• Patch on a previously upgraded version.


• Upgrade a previously installed-then-modified install.


• Patch a previously installed-then-modified install.

Posted by The Braidy Tester at 07:30 AM  Permalink