Dr. Dobb's | Crossing the Chasm

The agile cost-of-change curve, with new rhetoric and shorter feedback cycles, are all helping agile to further penetrate more traditional organizations.

Scott W. Ambler is a Canada-based software process improvement (SPI) consultant, mentor, and trainer with AmbySoft Inc. He has authored several books and is a regular speaker at software conferences worldwide.

Six years ago, if I'd told you that today's developers would be clamoring to improve their testing skills, to work closely with business stakeholders, to write high-quality code and to focus on providing business value whenever possible, you would have said that I was crazy. Well, that's exactly what we're seeing within the agile community. But many developers who wish to become agile are unfortunately still being held back by the organization within which they work. The good news? They won't be held back much longer. Agility is becoming mainstream.

One strong indicator of this change is how agile is doing on Geoffrey Moore's Technology Adoption Curve (Figure 1). Organizations on the left side of the curve are more interested in the potential of the idea and are willing to take a risk, whereas organizations on the right are very conservative and will wait for "solid proof" that the idea works in practice. The implication is that as a new technology or technique is adopted, its marketing approach must evolve to meet the expectations of the marketplace. Moore observed that the greatest change in the marketing approach occurs at the chasm—the organizations to the right of the chasm have significantly different expectations than those on the left. Many ideas fail in the marketplace because their supporters are unable to cross the chasm.

[Click image to view at full size]

Figure 1
Moore's technology adoption curve. Technologies and techniques are adopted at different times by different types of organizations.

As a concept, agile software development has clearly crossed the chasm—most Fortune 500 organizations that fall into the Early or Late Majority are at least talking about agile concepts and most are trying out various techniques. The Scrum methodology and techniques, such as refactoring and test-driven design (TDD), have definitely crossed the chasm. On the other hand, Extreme Programming (the XP name really doesn't help) has not crossed over yet, and database refactoring is so new that only the Innovators have caught on to it.

Why is Moore's adoption curve so important? Because we need a new sales pitch. To date, many within the agile community have sold techniques such as XP, Scrum, agile model-driven development (AMDD) and pair programming on their potential merits. That approach worked for the organizations on the left side, but it isn't working for those on the right side. Our target audience has changed, and so must our message.

First and foremost, the agile community needs to address the hyperbole. Our detractors will say, "Agilists don't write documentation." Nothing could be further from the truth. To further exacerbate the situation, some extreme agile evangelists claim that all the documentation is in the code. In reality, the goal is to store information only once, in the most appropriate place possible (see http://www.agilemodeling.com/essays/singleSourceInformation.htm). Even though a lot of the documentation is in the code, agile developers may often detail design information in unit tests, and requirement definitions in acceptance tests. But if the stakeholders insist on investing their time and money on traditional high-level models, system overviews and full support and user documentation, we will deliver them.

Other harmful rhetoric includes: "Agile teams don't need project managers, quality-assurance people or architects." Even though this is partially true, we still need aspects of all of these job functions. What we don't need are specialists who want to focus only on those specific roles. For instance, project management becomes a team function, but product managers are still needed to protect the team and obtain access to resources that are critical to project success. Quality assurance becomes a mindset that everyone on the team should have—after all, agilists are, by definition, "quality infected." Similarly, although architects are important to every system, architecture is something that we design in a highly collaborative and evolutionary manner, and we may not even have a need for comprehensive architecture documents (see http://www.agilemodeling.com/essays/agileArchitecture.htm for details).

But some of the rhetoric is absolutely brilliant: "We maximize the amount of work not done," is one of them. Agilists travel light. We've learned to discard traditionally comprehensive documentation for working software. When you travel light, you also reduce the bureaucracy surrounding those work products. For example, less documentation implies fewer documentation reviews and less need for traceability between documents.

The most interesting and controversial rhetoric is "the agile cost-of-change curve is flat." Once again, this is only partially true. In Extreme Programming Explained (Addison-Wesley, 2004), Kent Beck claimed that the cost-of-change curve for XP is flat, with the occasional bump in the production stage. The reason this claim is controversial is that the traditional cost-of-change curve is exponential—the longer you take to find a defect, the more expensive it is to fix. The reality is that agile methods occupy only the left-most part of the cost-of- change curve because agilists adopt techniques with very short feedback cycles (see Figure 2). This enables us to detect and address defects as early as possible. I explore this concept in detail at http://www.agilemodeling.com/essays/costOfChange.htm. The real message should be "Agilists follow techniques that minimize the cost of change."

[Click image to view at full size]

Figure 2
Comparing the cost-of-change "curves." The agile and traditional cost-of-change curves are identical. The difference is that agilists follow techniques that keep us at the low-cost end of the curve.

The primary strength of agile techniques is that they reduce the feedback cycle, and thereby the overall cost of software development. Figure 3 maps the effectiveness of agile (shown in green) with the traditional (in red) development techniques, onto the cost-of-change curve. Techniques such as Pair Programming and Modeling with Others have feedback cycles on the order of seconds, whereas test-driven design (TDD) is on the order of minutes and agile model-driven development (AMDD), on the order of hours. Compare these feedback cycles against traditional techniques, such as Reviews or Big Requirements Up Front (BRUF), which have feedback cycles on the order of weeks or months, respectively. I compare the feedback cycles of these techniques, and more, at http://www.ambysoft.com/essays/whyAgileWorksFeedback.html.

[Click image to view at full size]

Figure 3
Comparing the effectiveness of various development techniques.

Figure 3 is particularly important because it speaks to senior managers. Diagrams like this get their attention, and if the supporting evidence is persuasive, they might consider adopting some of the techniques. The Early Majority, in Figure 1, want to hear about anecdotal evidence; Figure 3 satisfies this need. However, since the Late Majority typically want to see several case studies of firms within their industry before they adopt a new approach, Figure 3 helps them to become more receptive to agile techniques. The Laggards want incontrovertible proof before trying something new: I suspect they'll still be sitting on the sidelines for a few more years.

My prediction is that in the next few years, we'll see agile techniques and even full-scale methodologies adopted within the Early and Late Majority organizations, and even among some of the Laggards. Although the current boutique consulting firms specializing in agile software process improvement will continue to thrive, we'll see a shift within the industry towards large, traditional consulting organizations (like Accenture or IBM Global Services) that have traditionally had the ears of senior IT management. Agile software development is clearly crossing the chasm, and so must the innovators within the agile community.