Process Improvement that Works

SIDEBAR: Process Improvement Resources

Many development organizations are climbing onto the software process improvement (SPI) bandwagon, but too many of them are falling back off again. To get a zero return on your SPI investment, follow this simple procedure:

• Spend money and time on process assessments, consulting services, and training seminars.
• Create a big honkin’ binder of procedures and tell all team members they must follow all of the procedures starting immediately.
• Hear senior management decree, “Make it so!”
• Watch the binders gather dust as team members never really change the way they work.

Here are some lessons from the process improvement trenches that can help you avoid this exercise in futility. I’ll share practical tips for how to approach SPI, along with descriptions of the basic process improvement steps.

What Is Process Improvement?

At its core, process improvement is simple: consistently apply the practices that give you good results, and change the practices that cause problems. This requires honest introspection and careful analysis to identify the reasons behind your previous projects’ successes and shortcomings. Your primary motivation should be to achieve specific business results through better software development and management approaches. You can use established process improvement frameworks like the Software Engineering Institute’s Capability Maturity Model for Software (CMM) to guide your efforts. Remember, though, your objective is not simply to satisfy the model’s expectations.

The Process Improvement Cycle

Figure 1 illustrates an overall SPI cycle. After defining your desired business objectives, assess your current processes, problems, and project outcomes. Armed with assessment insights and a knowledge of software industry best practices, you can then set realistic improvement goals. Select a small set of appropriate practices that will address your current process shortcomings and move you toward your goals. Identify a project or two on which to pilot new processes and make adjustments before officially rolling them out.

Figure 1: An Overall SPI Cycle

Planning how you will implement new ways of working greatly increases your chance of success. The hardest part is actually implementing an action plan; if you don’t, nothing will really change. Give the new processes some time, then see whether they’re easing the problems those improvement actions targeted. Hard data about process benefits is more convincing than subjective perceptions. Then continue the improvement cycle with the next most pressing need. Process improvement is a journey, not a destination.

Focus on the Pain

Pain is the best motivation for changing the way people work. I don’t mean external or artificially induced pain, but rather the very real pain we sometimes experience from our current methods. One way to encourage people to change is to persuade them that the grass will be greener in the intended future state. A more compelling motivation is to point out that the grass behind them is already on fire.

The assessment helps reveal the points of pain and major risks facing your projects. An assessment can be a simple brainstorming session in which your team members identify barriers to improving their productivity and quality. Alternatively, you can invest in a semi-formal evaluation by an external consultant, or in a rigorous formal process appraisal conducted according to an established process model such as the CMM. Such formal appraisals are expensive and time-consuming, but they thoroughly evaluate your current practices against some process standard.

In my experience, a process assessment doesn’t usually reveal major surprises. Most groups are already aware of their problem issues and behaviors. Having an external party conduct the assessment legitimizes it, because the external party is free of the development organization’s politics, historical tensions, and personalities. The assessment lets you confront the uncomfortable problem situations directly. An assessment should never be a forum for finding fault or blaming individuals for past problems.

Performing an assessment can also demonstrate management’s commitment to process improvement. Be sure to follow through on the assessment’s findings and recommendations. If you don’t, you’ll lose the money and time you’ve spent on the assessment, as well as credibility with team members.

An assessment usually identifies more improvement opportunities than you can tackle at once. “Focus” is a key word in process improvement. It takes longer than you probably think to devise better processes and make them part of the way your group routinely operates. I once saw a highly motivated, 20-person project team undertake seven improvement areas concurrently. Resources were spread thin, priorities were unclear, and the team accomplished little, despite the members’ enthusiasm.

State your process improvement objectives in terms of the desired business results. For example, a goal could be to eliminate incorrect builds being handed off from development to system testing, not merely to write a build-and-promote procedure. The SPI activities are a means to an end, not an end in themselves. Your managers should be able to articulate how they expect the group’s behaviors and results to be different if the SPI program succeeds.

Based on your top priority goals, pick two or three improvements to address initially. If you complete them quickly, great; go back to the assessment report and select the next area to work on. Don’t try to do too much right out of the gate. Large organizations can undertake several change initiatives concurrently across multiple projects, but each project team should focus on just a few areas at a time.

Communicate, Communicate, Communicate

People naturally resist changing the way they work, because they have a comfort level with familiar (if not always effective) practices and a fear of the unknown. Concern about burdensome process overhead that stifles creativity and delays delivery is also common, but the fear normally exceeds the reality. Your team members might also experience a sense of failure because the assessment identified process shortcomings, even though they’ve been working as hard as they can. Customers and other external groups might view the SPI program as raising barriers that will make it harder for them to get what they want from developers.

To address these issues, communication should permeate your process improvement activities. Articulate the price being paid for your current process shortcomings. This price might include blown schedules, missing functionality, massive overtime, high product-support costs, unhappy customers, or low morale. Explain to skeptics what benefits the improvement activities will provide to individuals, the project team, the company, and customers. Allay the fear of overwhelming change by stressing that new processes will be selected thoughtfully, created by the team members themselves, and rolled out gradually. Look for “allies” who are willing to try new procedures and document templates, provide feedback, and help lay the groundwork for successful change.

Trumpet your achievements to the team members and other stakeholders. Publicly recognizing the individuals who contributed to each successful change demonstrates that constructive participation in the SPI program is desired. Analyze unsuccessful change attempts to understand why they struggled and adjust your approaches accordingly.

A key SPI operating principle is “Gentle pressure, relentlessly applied.” Keep your improvement goals and status visible to the team. Stress how the SPI goals align with your organization’s business goals. Set aside time during team meetings to review the process improvement progress. Periodically report successes to management to bolster their support for the effort.

A kiss of death is to launch an improvement program with great fanfare at the beginning of the year, then never mention it again until you see whether or not you reached your goals at year’s end. You won’t. Working on project activities will always be more important to most team members than contributing to process improvement. Managers must frequently remind the team that SPI work is important and valued.

Organize for Process Improvement

Organizations that are serious about SPI often set up a three-level organizational infrastructure to make it succeed (as shown in Figure 2), but you should adapt these ideas to your organization. This structure shouldn’t be any more complicated than necessary to ensure that sensible improvement actions are identified, initiated, and implemented effectively. The management steering committee (MSC) commits resources and sets direction and priorities. It might also identify a “process owner” for each improvement area, a manager who is responsible for achieving improvements and provides continuity as individual working groups come and go.

Figure 2

Typical MSC members include the organization’s senior manager, the manager who is responsible for the SPI program, the individual leading the SPI effort, and selected project or department managers. Getting several management levels to participate actively in the MSC indicates that the organization is taking process improvement seriously. The MSC’s responsibilities include:

• Setting improvement area priorities.
• Chartering working groups to address specific improvement areas.
• Monitoring activities and status.
• Assessing the impact of the improvement actions completed to date.
• Managing process improvement risks and breaking down barriers.

The software engineering process group (SEPG) coordinates the various process improvement actions. The SEPG acts as management’s agents to implement the process improvement program. A large organization’s SEPG should have a full-time manager, some full-time software process specialists, and a rotating cast of practitioners who participate on a part-time basis for a year or so.

Process specialists often come out of the testing and quality assurance ranks, but several SEPG members should have substantial development experience. Project management experience is a needed perspective, also. These qualifications will give the SEPG more credibility with developers and managers than if it is viewed as a bunch of theoreticians who don’t know how real software development gets done.

SEPG members learn a lot about assessments, process improvement frameworks, writing good processes and procedures, and influencing change. The human and organizational issues of change management are at least as important as the technical aspects of process improvement. Effective SEPG members are good communicators who can adapt their approaches to individual situations. They are also well-organized, patient, and flexible. They are skilled facilitators who lead fruitful discussions on sensitive topics with diverse participants, some of whom probably don’t believe in SPI and would rather not participate.

The SEPG serves as a resource to all those involved in the process improvement program. The group’s expertise, resources, and outside connections accelerate the change initiative because practitioners involved in the effort have a place to go for help. SEPG members typically perform the following functions:

• Develop strategic and tactical improvement action plans.
• Lead or participate in process assessments.
• Coordinate and facilitate process improvement working groups.
• Collect and disseminate literature and information on industry best practices.
• Accumulate process assets, such as procedures, templates, checklists, and work product examples, and share them across the organization (see my article, “Improve Your Process with Online ‘Good Practices,’” Dec. 1998).
• Review new processes, procedures, and templates that working groups create.
• Lead improvement activities that span the entire organization, such as metrics and training programs.

It’s not a good idea to have the SEPG write all the new procedures and inflict them on the project teams. This classic “throw it over the wall to land on the victims’ heads” approach almost always fails. Instead, get practitioners involved in developing realistic new procedures through their participation in process improvement working groups, also called process action or process improvement teams. A working group is an ad hoc group of about three to six project representatives who address a specific improvement area. Their deliverables typically include a description of the processes currently being used in that area, as well as new processes and process assets. A working group’s efforts might lead to new processes for just one project, or they could develop processes for an entire organization.

Scope each working group’s objectives such that it can complete its tasks within about three months. If the working group goes on for too long, the members may lose their enthusiasm and turnover is likely. You can always re-charter it or convene a new group for the next round of improvement actions. Try to get all practitioners and project leaders involved in a working group at some point (although not all at once). This will help give all team members a sense of ownership toward the new processes. A member of the SEPG can launch each group and facilitate its initial meetings. However, it’s important for the development organization to adopt ownership of its improvement program and sustain its own working group efforts after the SEPG removes the training wheels.

Plan Your Actions

Treat your SPI program like a project, giving it the structure, resources, and respect that any development project needs. Two useful planning components are an overall strategic software process improvement plan and a tactical action plan for each working group. Some people balk at writing plans, viewing them as wasteful process bureaucracy and overhead. However, writing the plan isn’t the hard part. The hard part is thinking, asking, listening, understanding, negotiating, and thinking some more. Actually, writing the plan is mostly transcription at that point. I’ve found that plans help keep even simple projects on course and provide something against which I can track my progress.

Table 1 suggests a template for a strategic SPI plan, which guides the organization’s long-term improvement activities. The organization’s process improvement leader typically is the prime author of this plan, although members of the SEPG and key managers should review and approve it. Adapt this template to serve your own SPI planning needs.

Each working group should write an action plan, using a standard template, that describes what the group intends to accomplish and how. The action plan should identify:

• Business or technical goals to be accomplished, which should trace back to the overall goals expressed in the strategic SPI plan.
• Measures to tell if the process changes have had the desired effects.
• Organizational scope of these process changes.
• Participants, their roles, and their time commitments.
• How often and to whom the group will report status, results, and issues.
• Any external dependencies or risks.
• The target completion date for all activities.
• A list of action items, identifying for each the individual owner, date due, purpose, activities, deliverables, and resources needed.

Limit the number of action items in each plan to 9 or 10 specific, small actions. This will help you scope the working group’s efforts to just a few months.

Steer to Success

Process improvement leadership is a steering function, guiding the organization first to accept that better practices are needed and then to successfully implement them. This steering requires stable goals and constancy of purpose. Ever-changing improvement objectives confuse and frustrate practitioners, who might throw up their hands and say, “Tell me when you figure out what you really want.” Avoid being distracted by a quest for higher CMM levels; focus instead on improving your business results by selectively and creatively applying the guidance from existing frameworks such as the CMM.

Decide how serious you are about SPI and allocate resources accordingly. Organizations that spend less than 3% or 4% of their budget on process improvement (including training, assessments, consultants, SEPG staffing, and working groups) are dabbling. Organizations that invest 7% or 8% are pretty serious, and spending more than 10% of your budget indicates a major commitment. Track the time actually spent on SPI activities to see if the planned work is really getting done and whether your current resource level is consistent with your objectives.

Recognize the reality of the learning curve—the short-term performance drop that results as you learn new work methods and incorporate them into everyone’s personal processes. It will take time for individuals to internalize new and better ways of working, and for the organization as a whole to institutionalize new methods as routine practice. The time and money you spend on SPI is a strategic investment in the long-term success of your organization, and those resources aren’t available for immediate project use. Take satisfaction in small victories and celebrate your successes. Keep trying to do your project work better tomorrow than you did it yesterday, and you’ll come out ahead. That’s all process improvement is really about.