Email	Print Reprint
	add to:
	Del.icio.us Digg Google Furl	Slashdot Y! MyWeb Blink

Never do anything that's a waste of time—and be prepared to wage long, tedious wars over this principle," declares Michael O'Connor, project manager at Trimble Navigation in Christchurch, New Zealand.

O'Connor's product group at Trimble is typical of the "homegrown" approach to agile methodologies. I conducted a workshop there in August 2000, but O'Connor and his team had grown their own "light" methodology long before that time. O'Connor defines their process as "extremely" lightweight.

"Lightweight means no written design documentation. We tend to have written requirements and specifications, but we're currently looking to reduce the written material at this level, too. Lightweight means accepting that we can't identify and control every little task. Lightweight means that we don't submit much in the way of status reports. We tend to just try things rather than analyze them to death. Lightweight means we spend very little time on estimates." This group successfully delivers versions of their product—an embedded GPS controller system—on time.

While interest in agile methodologies has blossomed in the past two years, their roots go back at least a decade. Teams using early versions of Ken Schwaber's Scrum (see sidebar below), Peter Coad's Feature-Driven Development and my Adaptive Software Development were delivering successful projects in the early to mid-1990s. More recently, my encounters with two companies using agile approaches taught me a sense of why and when they work. In one case, a U.S. medical software application company completed a complex, three-year product line transition. In another, an Australian software development firm delivered a comprehensive bank lending system in Singapore.

First, however, I'd like to delve briefly into two questions that will help put these stories in perspective: What problem domain does agile development best address? And what differentiates agile development from rigorous or plan-driven methods?

Fit the Process to the Project
While agile practices—pair programming, iterative feature-driven cycles, customer focus groups—are applicable to nearly any project, I believe that there is a certain problem domain in which agile development improves the odds on success. I've labeled this problem category exploratory projects. Similar to projects searching out new oil reserves in the jungles of Borneo where the outcome is uncertain, exploratory projects are those in which the technology is either bleeding or leading edge, and the product or business requirements are unstable. The more volatile the requirements and the more experimental the technology, the more agile approaches improve the odds of success.

However, the technology dimension of exploration entails more than just computers and communications. I'm working with two companies that build sophisticated instruments—one in mass spectrometry and the other in geophysics—pushing the technology edge in biology, chemistry, material science and physics. These two companies are exploring in multiple dimensions at the same time, and rigorous, plan-driven methods are inadequate for their goals.

Concise, Chaordic Collaboration
The agile movement covers a broader set of issues than the word methodology connotes, so I use the word ecosystem to include the three characteristics that define agile development: a "chaordic" perspective, "collaborative" values and principles, and "barely sufficient" methodology. A chaordic (the word, a combination of chaos and order, was coined by Dee Hock, former CEO of Visa International) perspective arises from recognition and acceptance of increasing levels of unpredictability in our turbulent economy. Two concrete ramifications of managing in an unpredictable environment are that while goals are achievable, project details are often unpredictable; and that the foundation of many process-driven approaches, the goal of repeatable processes, is unattainable. In company after company, I've found successful projects that met the customer's vision, but in the end, looked nothing like the original plan. Furthermore, truly repeatable processes would be almost mechanical in nature, and no mechanical process could possibly react to the infinite variations that software projects encounter. The reason many projects attain their goals has little to do with repeatable processes and much to do with the skill and adaptability of the people who work on the project.

The second characteristic of agile development is "collaborative" values and principles. Organizations that employ rigorous methodologies don't value people less than agile organizations; they just view people, and how to improve their performance, differently. Rigorous methodologies are designed to standardize people to the process, while agile processes are designed to capitalize on each individual's and each team's unique strengths.

The concept of a "barely sufficient" methodology attempts to discover how much "structure" is enough. To be agile, one must balance flexibility and structure, and "barely sufficient" doesn't mean insufficient. Bare sufficiency reduces costs through streamlining, but even more importantly, it incorporates the chaordic perspective that creativity and innovation occur in a slightly messy environment, not a primarily structured one.

The two stories that follow, and the previous allusion to Trimble Navigation, illustrate these three characteristics. The teams employed short-term iteration plans, not trying to precisely predict an uncertain future. They relied on appropriately skilled staff and their collaborative interactions. And, finally, each of these successful projects used vastly streamlined processes.

From Vision to Reality: Medical Systems Transformation "It's a case where, if there is any such thing, we were almost too successful," says Debra Stenner, vice president of business planning and product development for IDX Systems Corporation in Burlington, Vermont, describing how IDX transformed their strategic vision into reality using Scrum.

Ken Schwaber, one of the original developers of the Scrum methodology, put me in contact with IDX and Stenner. Over the years, I've worked with a number of companies that have struggled to move a popular product from one technology platform onto another—it's usually been an ugly struggle, often lasting years longer than originally intended, and not infrequently ending in failure. IDX not only made the transition, but emerged with a much stronger product line to carry the company into the future. IDX, a medium-sized firm with $340 million in annual revenue, supplies the health-care industry with financial, administrative and clinical support software for group practices and academic medical centers.

In 1996, IDX began a five-year transition from vision to reality, using Scrum. The company had a successful product, but an aging technology platform: Digital Equipment (DEC) hardware, a MUMPS database and the MVS operating system. "Market research said we were never losing deals on functionality," says Stenner, "but losing them because of 1982-vintage technology." In addition, IDX's main product line—administrative and financial software—wasn't integrated either technologically or strategically with their radiology software. They needed a strategic vision that included both new technology and product integration, and they were overwhelmed at the prospect of implementing both concurrently. The transition wouldn't be easy, as it required three parallel but integrated projects. The first would rewrite the existing radiology information system to run on a new technology platform—Windows NT/2000, SQL, HTML, Visual Basic and Active Server Pages. The second project would build a suite of modules to interface with all the imaging equipment in the market—equipment from companies like GE and Siemens that had major market presence. The imaging applications involved additional new technologies for the team to learn. The third project would build the workflow engine.

"These were three big things," says Stenner. "Just the porting project was a huge job, plus we had to learn all this new stuff that we'd never done before in the imaging world. The plan was approved in August 1996, but we still had to wrap up existing projects and start staffing for the new projects.

"So when Ken Schwaber got involved in December 1996, I sat there, wide-eyed, with this great vision and opportunity, trying to figure out how to bring in key technical resources, how to wrap up our existing projects and how to get on with the new projects." In September 1997, after finishing their existing projects, IDX took the plunge with three new endeavors.

Scrum—named for the rugby term—was initially developed by Ken Schwaber and Jeff Sutherland, with later collaborations with Mike Beedle. Scrum provides a project management framework that focuses development into 30-day sprint cycles in which a specified set of back-log features is delivered. Scrum's core practice entails daily 15-minute team meetings for coordination and integration. Scrum has been in use for nearly 10 years, successfully delivering a wide range of products. (See Ken Schwaber and Mike Beedle's book, Agile Software Development with Scrum [Prentice Hall, 2002], or www.controlchaos.com and jeffsutherland.com/scrum/index.html for more information.)

"With Scrum, we were able to move pretty quickly in a couple of areas," says Stenner. "One, Scrum really gets you away from the traditional waterfall approach—with its long, drawn-out up-front processes." Instead, the entire development team was shipped off to customer sites. Stenner recognized that the team was having difficulty translating the vision into requirements and selecting what tasks to do first. "I think it's hard, almost a disadvantage, when you have a clean slate and new technologies, to figure out how to start. With no guidelines, it's overwhelming for a lot of people."

She then called six customers and sent the imaging team (this was the area in which requirements were new to IDX) to find out how the technology could improve their business processes. A week after the customer visits were completed, the combined team had finished a requirements document. From beginning to end, the requirements process took about a month. By the time they finished, everyone understood the requirements, according to Stenner.

From the requirements document, the team created their master backlog, a complete list of features from which each 30-day development sprint would be taken. The first team then started on a series of sprints to deliver functionality. The second team figured out how to chunk the existing product into modules, so they could port it to the new technology. Each team consisted of 10 to 12 people.

When I asked Stenner how the developers felt about Scrum, her reply was emphatic. "They love it. They really, really love Scrum. And I'll tell you why—they don't feel like they're always changing direction." In Scrum, once the work for the 30-day sprint has been decided upon, it doesn't change. Stenner told me that everyone felt they were able to concentrate, and had good team synergy. "They like the small, focused group. They like getting a list of things to work on and know that the list isn't going to be pulled out from under them. They like that by going to one 15-minute meeting each day, they aren't then pulled into endless other meetings."

By early 2000, the teams had been so successful in development that new sales began rolling in—which, in turn, destabilized development efforts. "For instance," reports Stenner, "last year [2000] our average salesperson was at 180 percent of quota, and some were at 400 percent. So the vision has been incredibly, phenomenally, successful."

Agile practices can often help companies be successful, but sometimes that success brings its own new set of problems. "We were almost too successful," Stenner says. "We needed to have done a better job of capacity planning for the installation and enhancement work."

Over the years, I've worked with a number of clients attempting to shift a product from one technology platform to another. Some succeeded, some didn't—but the difficulty of these types of projects is consistently underestimated by an order of magnitude. Who says you can't complete a three-year project in 30-day increments? IDX did just that.

Taming Monsters: The Singapore Banking Project
For those who wonder if agile approaches can scale up, Jeff De Luca, project director of Nebulon, an IT consulting firm in Melbourne, Australia, offers a couple of examples of Feature-Driven Development (FDD) project success. The first, a complex commercial lending application for a large Singapore bank, utilized 50 people for 15 months (after a short initialization period). The second project, for another large bank, employed 250 people over an 18-month period. "But I'd never do one of these [250-person projects] again," De Luca vowed. "It was way too big. A very large bank was trying to completely reengineer its technology platform."

I tracked De Luca down through Jon Kern, director of product development for TogetherSoft. I was looking for an FDD case story for my book, and subsequently spent several hours on the phone with De Luca and Paul Szego, one of his chief programmers. We also exchanged a number of e-mails.

FDD arose, in name at least, in the 199798 timeframe during the Singapore project. De Luca had been using a streamlined, light-process framework for many years. Peter Coad, brought in to develop the object model for the project, had been advocating very granular, feature-oriented development, but hadn't embedded it in any particular process model. These two threads—one from De Luca and the other from Coad—came together on this project to fashion what was dubbed Feature-Driven Development (see Peter Coad, Jeff De Luca and Eric Lefebvre's Java Modeling in Color with UML, Prentice Hall, 1999).

Feature-Driven Development (FDD) is a minimalist, five-step process that focuses on developing an overall "shape" object model, building a features list, and then planning-by-feature followed by iterative design-by-feature and build-by-feature steps. FDD's processes are brief (each is described on a single page), and two key roles in FDD are chief architect and chief programmer. (See www.togethersoft.com and www.nebulon.com for further information.)

The Singapore lending project had been a colossal failure. Prior to De Luca's involvement, a large, well-known systems integration firm had spent two years working on the project and finally declared it undoable. Its deliverables: 3,500 pages of use cases, an object model with hundreds of classes, thousands of attributes (but no methods) and, of course, no working code. The project—an extensive commercial, corporate and consumer lending system—incorporated a broad range of lending instruments (from credit cards to large multi-bank corporate loans) and a breadth of lending functions (from prospecting to implementation to back-office monitoring). "The scope was really too big," said De Luca.

Less than two months into the "new" project, De Luca's team was producing demonstrable features for the client. The team spent about a month working on the overall object model (the original model and what he refers to as the previous team's "useless cases" were trashed), and then the team spent another couple of weeks working on the feature decomposition and planning. Finally, to demonstrate the project's viability to a once-burned and skeptical client, De Luca and his team built a portion of the relationship management application as a proof of concept.

From this point on, with about four months elapsed, they staffed to 50 people and delivered approximately 2,000 features in 15 months. The project was completed significantly under budget, and the client, the bank's CEO, wrote a glowing letter about the project's success.

Paul Szego filled me in on some of the details. "We often talk about 'self-organized' construction within planned assembly. There are a couple of levels of delegation. At the top, the project manager and/or development manager look at what features they want started next. Often this is done in whole business-activity chunks, but sometimes they select individual features to schedule. Each feature is assigned to the chief programmer responsible for that business activity."

"Each chief programmer has this 'virtual inbox' of features. The chief programmer knows what features are currently being developed, what's planned to happen over the next weeks and which programmers are tied up doing what tasks. They decide what's to happen next, and formulate work packages."

"In contrast, the project manager is more focused on the overall project plan—what things we do in what sequence, at a macro level. They bring risk forward by tackling more complex tasks earlier, satisfy any external constraints like interim deliverables or demos, and, of course, manage the planned dates for each business activity."

As De Luca, Szego and I talked, a couple of things struck me about this project. Certainly the FDD process contributed to the project's success. When I asked De Luca what made FDD successful, though, his first response was that the overriding assumption behind the process is that it embraces and accepts software development as a decidedly human activity. The key, he said, is having good people—good domain experts, good developers and good chief programmers. No process makes up for lack of talent and skill.

Based on his successful delivery of several complex applications, I'd say that De Luca is a very talented and skilled project manager. For his part, he stated that Peter Coad was a world-class object modeler, and that Coad indicated that this was the most complex domain he'd ever modeled. It appears that this lending system was a world-class problem, and that the ultimate critical success factor was assembling world-class talent. We can assume that the original vendor assigned reasonably talented and skilled people to the project and that it employed a "proven" process—but it didn't have world-class talent assigned to the effort. De Luca recruited a few highly talented people for key architect and chief programmer roles and then used these people to train and mentor others.

I suspect that even if the first vendor's staff had used FDD as a process model, they would not have been successful because they just didn't have the appropriate level of technical and project management talent. However, had they been using an FDD-like agile process, their inability to complete the project might have become evident in less than two years. Only working code is the ultimate arbiter of real progress, thousands of use cases and hundreds of object model elements notwithstanding.

The Bottom Line
Throughout the last two years, I've used agile methods with project teams in India, Canada, Norway, the U.S., New Zealand, Poland and Australia. These companies are struggling with exploratory projects that run the gamut: an e-commerce infrastructure product, a clinical drug-trial monitoring application, 300,000 lines of embedded C code in a new cell-phone chip, a worldwide financial system product, a myriad of internal IT applications, the complete business system for a dot-com start-up (one that's still in business), and an oil-field geophysical data gathering and analysis system.

These companies want to be more agile: They want to create change for their competitors and respond quickly to market conditions. They plan, but they aren't blinded by those plans. They focus on delivering customer value, not on adding up how many processes they have in place. They document, but they don't get lost in piles of paper. They rough out blueprints (models), but they concentrate on creating working software. They focus on individuals and their skills, and on the intense interaction of development team members among themselves and with customers and management. In short, they deliver results in a turbulent, messy, ever-changing, ever-exciting marketplace

RELATED ARTICLES Extending Enterprise Value with Web 2.0 .NET Development & the IBM WebSphere Portal Server Signalling Integer Overflows in Java Managing Application Thread Use The Android Mobile Phone Platform		TOP 5 ARTICLES No Top Articles.