Enhancing the Unified Process

Ask a senior executive within your organization to be candid about what he or she thinks about the work being produced by software professionals, and the answer may surprise or even disturb you. You’re likely to hear that your software projects are usually late, over budget, likely to be cancelled, and rarely meet expectations. The reality is that our industry has an 80% to 90% failure rate at large-scale, mission-critical projects (Pattern of Software Systems Failures and Success by Capers Jones, International Thompson Computer Press, 1996). It is clear that we need to change our ways, and an important start is for organizations to adopt a more mature software process.

A software process is a set of project phases, stages, methods, techniques, and practices that people employ to develop and maintain software and its associated artifacts (plans, documents, models, code, test cases, and so forth). You must combine the best of breed from each of the three leading processes—the Unified Process, the OPEN Process, and the Object-Oriented Software Process—to meet your organization’s exact needs.

An effective software process will help your organization increase productivity in several ways. First, by understanding the fundamentals of how software is developed, you can make better decisions when purchasing tools, hiring new employees, and managing your software portfolio—your collection of software projects currently in proposal, development, or production. Second, a software process lets you standardize and organize your software efforts, promoting reuse and consistency between project teams. Third, it lets you introduce industry best practices such as code inspections, configuration management, change control, and architectural modeling to your development staff. Fourth, it improves your organization’s maintenance by defining how to manage change and appropriately allocate maintenance changes to future releases of your software. And finally, it improves your support processes by specifying how to transition software into production and then how to operate and support it once there—without effective operations and support processes, your software will quickly become shelfware.

Software Process Requirements

Just as you start a software project by defining its requirements, you should also start discussing a software process by defining its requirements. The Software Engineering Institute (SEI) at Carnegie Mellon University has proposed the Capability Maturity Modeling (CMM), a framework for defining a process for large, complex software efforts. The CMM defines five process maturity levels, described in Table 1, evolutionary plateaus toward achieving a mature software process. To achieve a specific maturity level, an organization must satisfy and institutionalize all of its key process areas, described in Table 2. Sponsored by the U.S. Department of Defense, the CMM has been adopted by hundreds of organizations worldwide.

As your software process maturity grows, the quality of the software that it produces increases, a direct result of the increased focus on management and project deliverables. Further, risk on your software development projects is reduced, the result of increased management control and use of measurements (metrics) to understand and manage the software process. Most important, mature software organizations have a higher success rate and a lower overall cost of software than software organizations without an established process.

Although the SEI CMM focuses on software development, it also touches on maintenance and support. Before delving into the three specific processes, it’s important to note that development is a small portion of the overall software life cycle. The relative software investment that most organizations make is allocating roughly 20% of the software budget for new development, and 80% to maintenance and support efforts.

The Unified Process

The Unified Process is the latest endeavor of Rational Corp., which introduced the industry-standard, object-modeling notation, the Unified Modeling Language (UML). The heart of the Unified Process is the Objectory Process, one of several products and services that Rational acquired when it merged with Ivar Jacobson’s company, Objectory, several years ago. Rational enhanced Objectory with its own processes, and those of other tool companies, that it has either purchased or partnered with, to form the initial version (5.0) of the Unified Process, released in December 1998, encapsulated in the Rational Suite 1.1, the latest bundle of Rational tools.

Figure 1: The Unified Process Life Cycle

Figure 1 presents the life cycle of the Unified Process, comprised of four serial phases and nine core workflows. Along the bottom of the diagram, you see that any given development cycle through the Unified Process is organized into what Rational calls iterations. Basically, at the end of each iteration you produce an internal executable that your users can work with. This reduces your project’s risk by improving communication between you and your customers. Another risk reduction technique used in the Unified Process is to make a go/no-go decision at the end of each phase—if a project is going to fail, then you should stop it as early as possible in its life cycle.

The inception phase is where you define your system’s project scope and business case. Basic project management documents are started during the inception phase, including the initial risk assessment, the estimate, and the project schedule. The elaboration phase focuses on analyzing the problem domain and defining an architectural foundation for your project. You develop a detailed project plan for the construction phase, based on the initial management documents started during inception, in this phase. The construction phase is where you develop the detailed design for your application, as well as the corresponding source code. The goal of this phase is to produce the software and supporting documentation to be transitioned to your user base. The transition phase’s purpose is to deliver the system to your user community.

The Unified Process has several strengths. First, it is based on sound software engineering principles such as an iterative, requirements-driven, and architecture-based approach to development. Second, it provides several mechanisms, such as a working prototype at the end of each iteration and the go/no-go decision point at the end of each phase. Third, Rational continues to make a significant investment in its Rational Unified Process product, an HTML-based description of the Unified Process that your organization can tailor to meet its exact needs. At the time of this writing, the current version is 5.1.1 and version 5.5 should be available when this article goes to print.

The Unified Process suffers from several weaknesses. First, it is only a development process. As shown in Figure 1, it misses the concept of maintenance and support. Second, the Unified Process does not explicitly support multi-project infrastructure development efforts, such as organization-wide architectural modeling (see “Object-Oriented Architectural Modeling,” Thinking Objectively, Sept. and Oct. 1999), missing opportunities for large-scale reuse within your organization. Third, the process is likely not yet sufficient for the complex needs of modern developers because Rational’s approach to developing it was initially tools-driven. Can you automate every aspect of the software process? Is the Rational product offering truly comprehensive? These are questions that you need to ask when evaluating the Unified Process.

The OPEN Process

The OPEN consortium (www.open.org), a group of individuals and organizations promoting and enhancing the use of object-oriented technology, has developed the OPEN Process, a comprehensive software process that competes with the Unified Process. The OPEN Process is aimed at organizations using object and component technology, although you can easily apply it to other software development technologies. Similar to the Unified Process, OPEN was created by merging earlier methods: MOSES, SOMA, Firesmith, Synthesis, BON, and OOram. It supports UML, Object Modeling Language (OML), and other object-oriented notations to document your products. The shape of the lines and bubbles may change, but the modeling fundamentals remain the same.

Figure 2: The OPEN Contract-Driven Life Cycle

OPEN process’s contract-driven life cycle, shown in Figure 2, takes a traditional “bubbles and lines” approach, rather than the “humpback” life cycle of Figure 1. Each OPEN activity is shown as a rectangle, a similar concept to the workflows in the Unified Process, with unbounded activities shown as rounded rectangles and tightly bound activities as simple rectangles. The activities of the OPEN Process are subject to stated contracts, making OPEN a responsibility-driven process. Activities describe the overall process architecture; these are complemented by small-scale, project management tasks. Both activities and tasks describe what is to be done and a set of techniques describes how it is to be done.

Unlike the Unified Process, the OPEN Process life cycle explicitly includes activities outside the scope of a single project. This is called “programme management” in the OPEN Process, a programme being a collection of projects or releases of an application or suite of applications. Common programme management activities include organization-wide architectural modeling efforts, process improvement efforts, and standards or guideline development and support. Software organizations often call this sort of effort enterprise management or infrastructure management.

The OPEN Process benefits from several strengths. It is currently the most comprehensive of the three processes presented in this article, including a cradle-to-grave approach to a project life cycle and a multi-project view to software that reflects the actual environment of most organizations. It is also the brainchild of a wide variety of practitioners and academics, all of whom are coming to the table with different experiences, skillsets, and backgrounds. This breadth of talent is one reason why the OPEN Process is comprehensive and has resulted in a smorgasbord of development techniques from which to choose. And finally, it isn’t constrained by what can be automated or what can be easily marketed to an existing customer base: the members of the OPEN Consortium are free to call it as they see it.

The OPEN Process does suffer from one serious drawback: ineffective marketing. Remember the ill-fated Object Modeling Language (OML), the notation that competed against the UML several years ago in the race to become the industry standard notation? The OML was superior in many respects to the UML, but it didn’t garner the marketshare that the UML did. My fear is that once again the best candidate will be out-marketed by the second best.

The Object-Oriented Software Process

Figure 3: The Object-Oriented Software Process (OOSP)

Figure 3 depicts the Object-Oriented Software Process (OOSP) life cycle, comprised of a collection of process patterns. A process pattern is a collection of general techniques, actions, or tasks (activities) that solve a specific software process problem taking the relevant forces and factors into account. Just like design patterns describe proven solutions to common software design problems, process patterns present proven solutions to common software process patterns. Process patterns were originally proposed by James Coplien in his paper “A Development Process Generative Pattern Language” in Pattern Languages of Program Design, edited by Coplien and Douglas Schmidt (Addison-Wesley, 1995).

There are three scales of process patterns: phase, stage, and task. A phase process pattern depicts the interactions between the stage process patterns for a single project phase, such as the initiate and deliver phases of the OOSP. A stage process pattern depicts the tasks, which are often performed iteratively, of a single project stage such as the program and model stages. Finally, task process patterns address lower-level process issues, such as the Reuse First process pattern that describes how to achieve significant levels of reuse within your organization and the Technical Review process pattern that describes how to organize reviews and inspections. The three scales of process patterns are conceptually similar to the scale of other types of patterns: in the modeling world, you have a hierarchy (from most to least abstract) of architectural patterns, design patterns, and programming idioms.

An important feature of a process pattern is that it describes what you should do but not the exact details of how you should do it. Process patterns are an excellent mechanism for communicating approaches to software development that have proven to be effective in practice. When applied together in an organized manner, you can use them to construct a software process for your organization, as shown with the OOSP life cycle in Figure 3. Because process patterns do not specify the details of how to perform a given task, you can use them as reusable building blocks from which to tailor a software process that meets the specific needs of your organization.

In Figure 3, there are four project phases within the OOSP—initiate, construct, deliver, and maintain and support—each of which is described by a corresponding phase process pattern. It also has 14 project stages: justify project, define and validate initial requirements, define initial management documents, define infrastructure, model, program, test in the small, generalize, test in the large, rework, release, assess, support, and identify defects and enhancements—each of which is described by a stage process pattern. Project stages are performed iteratively within the scope of a single project phase. Project phases, on the other hand, are performed serially within the OOSP. This is a similar concept to the Unified Process, as shown in Figure 1, which is comprised of four serial phases and nine workflows that you perform iteratively.

There is more to the OOSP than just its phases and stages. The “big arrow” at the bottom of the diagram indicates important tasks that are critical to a project’s success and applicable to all development stages. These tasks include quality assurance, project management, training and education, people management, risk management, reuse management, metrics management, deliverables management, and infrastructure management. Several of these tasks are applicable at both the project and cross-project (programme) levels. For example, you should perform risk management for both a single project and for your software project portfolio—while each project can be relatively risk-free, as a collection they can become quite risky. Deliverables management, which includes configuration management and change control functions, is also applicable for both a single project and for a collection of projects, because the collection may share a single artifact, such as common source code. Infrastructure management—where you evolve your corporate processes, standards, guidelines, and architectures—is applicable across several projects or a single project because that project team will need guidance and support to use your organization’s software infrastructure.

The process patterns of the OOSP reflect the experiences of a large number of object practitioners, including the software professionals of Toronto’s Mark Winter & Associates, St. Louis-based Insight Technology Group, and Denver-based Ronin International. These process patterns reflect the lessons learned from a multitude of small, medium, and large projects in a wide range of industries, including telecommunications, finance, Internet, system outsourcing, government, military, and manufacturing. Like the OPEN Process, the OOSP combines real-world experience with proven software engineering theories.

There are several strengths to the OOSP. First, it is a comprehensive approach to the software process, taking the entire life cycle into account. Second, it accurately portrays that large-scale, mission-critical software is actually serial in the large and iterative in the small—contrary to popular belief, real-world development isn’t purely iterative. Third, like the OPEN Process, it explicitly supports programme-infrastructure processes, enabling large-scale reuse and effective management of your software projects portfolio. Fourth, significant work is being done in process patterns and organizational patterns, with an evolving body of knowledge posted online. Fifth, the OOSP life cycle explicitly includes maintenance and support. Finally, the OOSP explicitly includes things that have a tendency to fall through the cracks, such as generalizing your work to make it reusable, performing risk management, performing quality assurance, and assessing your project and project team once your software has been delivered.

There are also two weaknesses. First, because the OOSP is comprehensive, it can be overwhelming to most software professionals, who prefer to focus on their small parts of the software process. Second, like the OPEN Process, the OOSP suffers from similar marketing challenges.

Comparing the Leading Processes

Earlier I stated that the Capability Maturity Model (CMM) effectively forms the requirements for a development process, so let’s start the comparison of the three processes there. Table 2 lists the CMM’s key process areas, indicates which CMM level each area applies to, and then provides my rating of how well each process implements each area. The OOSP supports these key process areas the most thoroughly, which should be expected because the CMM was used as the requirements model for that process. The OPEN Process also fares well because of its strong ties to the CMM and the process community in general. The Unified Process does not fare as well, likely because it was melded from various tool-based processes instead of engineered from first principles.

The OOSP and OPEN Process also fair better than the Unified Process, for the most part, because they cover the entire software process and not just the development process. They also support programme-infrastructure activities, whereas the Unified Process focuses on a single project. The Unified Process, however, fares better in its explicit support for iterative development and in its mindshare within the marketplace.

You’ve seen how they compare, but which should you choose? I believe the answer is the Unified Process. Although I definitely have skin in the game with the OOSP, I have to recognize that, for better or worse, the Unified Process is the market leader and will likely remain so despite its challenges. The Unified Process isn’t perfect—none of them are—but it’s the one that we need to make work as an industry if we want organizations to be successful at software.

Completing the Unified Process

So how do you complete the Unified Process? Start by rescoping it to include the entire software process, not just the development process. This implies that you must add operations, support, and maintenance processes. Second, the Unified Process also needs support for managing a project portfolio. These first two steps result in a reworked version of the life cycle, which I call an enhanced Unified Process life cycle, shown in Figure 4. Finally, the Unified Process needs to be enhanced with material from both the OPEN Process and the OOSP to fill the remaining gaps.

Figure 4: An Enhanced Unified Process Life Cycle

The enhanced Unified Process includes a fifth phase, production. As the name implies, the purpose of this phase is to keep your software in production until it is either replaced with a new version, from a minor release (such as a bug fix) to a major new release, or it is retired and removed from production. Note that there are no iterations during this phase, or only one depending on how you look at it, because this phase applies to the lifetime of a single release of your software. To develop and deploy a new release of your software, you must run through the four development phases again.

Figure 4 also shows two new workflows, a core workflow called operations and support and a supporting workflow called infrastructure management. Operations and support are both complex endeavors that require defined processes. During the construction phase, you will need to develop operations and support plans, documents, and training manuals. During the transition phase, you will continue to develop these artifacts, reworking them based on the results of testing, and you will train your operations and support staff to effectively work with your software. Finally, during the production phase, your operations staff will keep your software running, performing necessary backups and batch jobs as needed, and your support staff will work with your user community in working with your software. This workflow basically encompasses portions of the OOSP’s release and support stages as well as the OPEN Process’s implementation planning and use of system activities. In the Internet economy where you do operations nonstop, you quickly discover that high-quality and high-availability is crucial to success—you need an operations and support workflow.

The infrastructure management workflow focuses on the activities required to develop, evolve, and support your organization’s infrastructure artifacts such as your organization-wide models, software processes, standards, guidelines, and reusable artifacts. Your software portfolio/programme management efforts are also performed in this workflow. Infrastructure management occurs during all phases, the bump in the elaboration phase represents architectural support efforts to ensure that a project’s architecture appropriately reflects your organization’s overall architecture. In “Architectural Modeling” (Thinking Objectively, Sept. and Oct. 1999), I describe a modeling process that includes infrastructure modeling activities such as the development of an enterprise requirements or business model, a domain architecture model, and a technical architecture model. These three core models form your infrastructure models that describe your organization’s long-term software goals and shared or reusable infrastructure. The processes followed by your Software Engineering Process Group (SEPG)—which is responsible for supporting and evolving your software processes, standards, and guidelines—are also included in this workflow. Your reuse processes are also included—practice has shown effective reuse management to be a cross-project endeavor. The infrastructure management lets you achieve economies of scale developing software, to increase its consistency and quality, and to increase reuse between projects.

Comparing the enhanced Unified Process of Figure 4 with the Unified Process of Figure 1, you will notice that several of the existing workflows have also been updated. First, the test workflow has been expanded to include activity during the inception phase. You develop your initial, high-level requirements during this phase that you can validate by using techniques such as walkthroughs, inspections, and scenario testing. The test workflow also needs to be enhanced with the techniques of the OOSP’s test in the small and test in the large stages (see “Object-Oriented Testing,” Thinking Objectively, July 1999.)

The second modification is extending the deployment workflow into the inception and elaboration phases. This modification reflects that deployment, at least of business applications, is a daunting task. Data conversion efforts of legacy data sources are often a project in their own right, a task that requires significant planning, analysis, and work to accomplish. Further, deployment modeling should be part of the deployment workflow and not the analysis and design workflow as it currently is, because deployment modeling and deployment planning go hand in hand. Deployment planning can and should start as early as the inception phase and continue into the elaboration and construction phases, simultaneously with deployment modeling.

The environment workflow has been updated to include the work necessary to define the production environment. The configuration and change management workflow is extended into the new production phase to include the change control processes needed to assess the impact of a proposed change to your deployed software and to allocate that change to a future release of your system. Similarly, the project management workflow is also extended into the new production phase to include the processes needed to manage your software once it has been released.

The project management workflow is expanded in the enhanced Unified Process. As mentioned earlier, this process is light on metric management activities. It also needs processes for subcontractor management, a CMM level 2 key process area, a key need of any organization that outsources portions of its development activities or that hires consultants and contractors. People management issues, including training and education as well as career management, are also added—there is far more to project management than the technical tasks of creating and evolving project plans; you also need to manage your staff and mediate the interactions between them and other people.

Choosing Maturity

Developers have known the fundamentals of the software process for years. But as an industry, we have generally ignored these fundamentals in favor of flashy new technologies that promise to eliminate all the inherent complexities of our jobs—resulting in a consistent failure rate of roughly 85%. Added to the embarrassments of the Year 2000 crisis, this failure rate is a clear sign that we need to change our ways. It is time for organizations to choose to be successful, to follow techniques and approaches proven to work in practice, and to follow a mature software process.