Metrics Business Case
Metrics can provide information necessary to understand the types of process changes that can improve the quality and reduce the cost of a given project and provide a clear indication of the level of improvement as a result of process changes. Using the five points above can help charter the way to instituting a metrics program and effectively monitoring the benefits of process improvement initiatives.
The following case study demonstrates the successful execution of test metrics within a real-life company setting.
Background
The IT department of a major truck manufacturer had little or no testing across its IT projects. The company's projects were primarily maintenance related and operated in a COBOL/CICS/Mainframe environment. The truck manufacturer wanted to migrate to more up-to-date technologies and felt that testing should accompany this technological shift. The company needed to establish a testing process and also train new test team members.
Course of Action
The test team was introduced to test metrics it should track. The team's first project, Project V, was primarily developed in Visual Basic and HTML, and was accessed via standard web browser technology. By the time the test team became involved in the project, all of the analysis and most of the development had been completed. The test team developed 355 test cases and had a 30.7 percent first run failure rate along with an overall failure rate of 31.4 percent.
Believing that earlier test team involvement would improve the development process, with the company's second project, Project T, the test team instituted requirements and specifications walkthroughs. Project T was slightly more complex in that it added XML to the new development environment of Visual Basic and HTML. Project T required 345 test cases and used substantially the same staff as Project V.
Results
The first run failure rate for Project T was 18.0 percent, and the overall failure rate was 17.9 percent -- dramatically better than the results from Project V. By reducing the overall Failure Rate, the accumulated costs of rework were also reduced, creating a cost saving of approximately $170,000.
When managing a test effort, test leads and test managers sometimes find it difficult to empirically convey to the project manager, project team and other interested parties, the impacts of scope changes, delays and defects. Consistently applying a set of well-defined test metrics to track and manage a test effort can dramatically improve the ability to effectively and objectively communicate findings across an organization.
