Practicing .NET

Improving developer productivity and software quality

January 2007

Last time I mentioned I had looked at Agile/XP or Extreme Programming a few years ago and although I was keenly interested in its approach to software development, I couldn't get past the hurdles in introducing it into an existing group of developers. Too much of a culture shift, and I didn't see how it would immediately help with coordination with other teams such as the user interface design group, QA, Program Management, and so on.

And so my interest in Agile stayed pretty much that. Once in a while I'd hear about other Agile approaches but my mental model was locked into the ideas around XP and so I would wander on to other technology topics. However, time passed and the realization that the old-style "waterfall" model of software development (the "if only enough time is spent at each stage, all will be well" philosophy) was interfering with our ability to react faster to market and internal demands.

So I started my research again into Agile practices to see what had changed, or what else was available that might help. Not too long into the reading and learning phase, I happened upon the Agile process that has come to be known as Scrum. Odd name, I thought. Not quite as hip sounding as "XP". But for whatever fortuitous reason, I lingered a bit longer and read up on Scrum and its approach to Agile.

Boy, am I glad I did.

I got so enamoured with the holistic yet straightforward approach of Scrum to software development that I attended a Certified Scrum Master course taught by CC-Pace in Fairfax VA. It was a great experience to say the least. Small class, lots of interaction, many different experiences with software development and even a few Agilists (XP'ers). There was a great deal of role playing and working through software development scenarios. At the end, after passing a test certification was granted. Now, being a certified Scrum Master doesn't by any means state you are a master of Scrum. That will take years of study and application to learn the nuances of such a simple approach to software development. Instead, it means that you are certified to play the role of a member of the team designated as the ScrumMaster - in essence, the person who guides the team in all aspects of Scrum. However, the ScrumMaster is not actually a member of the team doing the actual work. And this is a key point in understanding Scrum.

Unlike XP, Scrum positions itself as a "work management process" rather than a "software development process". It can be used on non-software efforts including managing a business, managing services, and so on. It's a way to organize a team of cross-functional people to "deliver priortized business value every 30 days". Wow, sounds like a mouthful.

Actually, Scrum is straightforward and that was a key reason I liked it. It's basic principle is items of work are completed in a priority order (determine by a Product Owner) by a team of people (the Team) every 30 days. The 30 days can be changed under local circumstances to 15 days, but 30 days has been viewed as normal practice by results from the 15 years that Scrum has been in use. Here's how it works:

On day 1 of the cycle (called a Sprint), the Team meets with the Product Owner and the Scrum Master to go over a prioritized backlog of items (called the Product Backlog). These are essentially the work items that need to be done without regard to difficulty to implement or complete them. The Product Owner applies whatever business reasons they need to in order to come with the priority. The Team then goes down the list one-by-one asking questions about each item.

Once an item is understand at a reasonable level, the Team conducts a group estimate of the effort needed to complete the item. The estimate is done by voting on a weight (something developers do all of the time when they say this is small, medium, large). Votes are taken in succession until the team coalesces around a number that they can agree on. This then becomes the number or weight of that item. This process continues until the Team estimates more work than they can likely complete in 30 days. This set of work forms the backlog items for the Sprint. Interestingly, the meeting itself has a time-box of 4 hours to keep it from becoming interminable. Afterwards, the Team leaves to develop a set of tasks for each item and estimates each of those as remaining-hours-of-work. Again, this step is time-boxed to 4 hours.

On day 2, the work actually begins.

Next time, I'll continue with the look into how Scrum works.

Technorati Profile

Posted by Mark M. Baker at 03:17 PM  Permalink |

I've been talking for a bit now about Agile and XP software development practices. I mentioned that many developers first learn about Agile practices by reading Kent Beck's series on Extreme Programming (XP) or stumbling across the myriad of articles on the Internet about Agile/XP. For myself, this is how I first got introduced to Agile. I bought a series of XP books, and plunged into some long nights reading and digesting the revolutionary ideas around Agile.

However, like most revolutions, they force drastic changes in established practices that people follow. Sometimes this is necessary. Especially in software development, the practices of the last 20-30 years pushed by project management experts that if we only spend enough time on requirements and design all will be well, have been discredited by emperical evidence (late projects, failed projects, etc) and needs to change. Clearly what we've been doing hasn't worked too well in the general case. But getting people to willingly change what they're comfortable with is a significant challenge.

Although I could see how XP practices could help a development team get software written faster, I was hesitant to adopt it for the following reasons:

Developer resistance to Pair Programming
A key method in XP to increase the quality of generated code is to double up on the implementation by pairing developers. This includes pairing for the architecture, design and eventual coding. For certain profiles of developers, this can work really well and there are groups for which it does. However, there are also excellent developers that resist the notion of writing software this way. Although the true believers might state "then you need new developers", for many projects this is simply not an option. People accumulate knowledge about software over time, and companies cannot afford to be cavalier about seeing that knowledge walk out the door. The problems mount for any development manager if the developers who are already resisting this aspect of XP, have personality conflicts when working in such close quarters with their team mates. The question for any development manager here is whether the gains from XP Pair Programming outweight the inevitable turmoil it will cause to the existing team especially a team that is new to Agile in the first place.

Lack of Cross-Functional Emphasis
XP is definitely an Agile practice. However, it was created by developers and focuses on problems developers have in creating software. Other members of the software team such as Quality Assurance, User Education/Documentation, User Interface Design, DBA, etc are not specifically included in the XP framework. They can be fit in, but how they fit in isn't stipulated by XP. On projects I've worked on, the developers may have a huge say in how the software is built, but they rarely get to say when it is shipped (that's QA), how to present it to users (that's UI design), or how it's described (that's User Education). The software isn't complete until these other parts of it are also complete.

I had some other concerns around how to do Test Driven Development when maintaining large applications that have no history of unit tests nor an easy way to create them due to the particulars of the technology. But this was a lesser concern compared to the others I mentioned earlier.

So I put Agile/XP on the "interesting, but in a holding pattern" part of my bookshelf. And there it sat for quite a while, years in fact. Every now and then I would return and speculate on ways to introduce Agile/XP to the team without destroying it in the process.

And then I stumbled upon another Agile process called Scrum.

Next time, I'll talk more about what I like about Scrum and why it fits better for me than a pure XP approach does.

Technorati Profile

Posted by Mark M. Baker at 03:27 PM  Permalink |

So I mentioned last time the principles that XP brings to software development specifically the ways in which developers approach working on an application. If there's an overarching, elevator pitch statement about XP, I would put it as this:

XP (and Agile in general) focuses on delivering known value today rather than rumored value tomorrow.

The trick is how developers can accomodate the fast turnarounds of small chunks of functionality that need to work (and work well) vs just be at some random level of completion. Gone are the lengthy amounts of time spent doing exhaustive analysis and design, test cycles, and so on at distinct phases of a development lifecycle. They are replaced by doing many of these things simultaneously.

By observing what is good in software development (or maybe better put as what works), Kent Beck came up with some (paraphrased) observations:

If testing is a good thing, then test all the time.

If integrating code is a good thing, then integrate continuously.

If simplifying code is a good thing, then re-factor it all the time.

If working together to implement a solution is a good thing, then work together all the time.

Testing all the time means that we focus on treating tests as co-equal to the code. They are not things that we write haphazardly at the end, or not at all. Tests are written up front before the code is actually written. Code is not assumed to be working until the tests pass. As code is changed, tests are constantly re-executed checking for problems.

Integration all the time means that we focus on building the entire application at least daily and preferably whenever a code check-in is performed by a developer. Constant integration ensures we know immediately if there are code units which have changes which have broken other parts of the system. Ideally, integration is done automatically by a centralized build machine without developer intervention.

Simplifying (refactoring) code all the time means that developers are looking for ways to remove code, not just add it as features are developed. A corollary to this is that removing code should be done all the time. Most systems have huge amounts of seldom-used code that can be refactored to do the same with less effort. The key is to have a robust set of tests to ensure the refactoring doesn't change the semantics of the code.

Working together all the time means that developers pair up to write code rather than once in a while. This includes everything around writing code including data design and architecture. This also has the added benefit of spreading the knowledge around so the team isn't dependent on any one developer exclusively.

Particularly if you are working on a new project, the XP model for software development is very attractive. Software gets built faster with more focus on the items most relevant to the customer and with more reliability. The challenge of course, is what to do when you're working on projects which are pre-existing, have no body of unit tests, are under constant time pressure, knowledge is in silos, the team is distributed, and so on.

For the answer to that, next time I'll explain why I chose Scrum rather than pure XP as my initial Agile software development model.

Technorati Profile

Posted by Mark M. Baker at 04:33 PM  Permalink |

I mentioned last time that my teams are now using Agile practices in our software development efforts. When you say something like that to another developer (or development manager) a range of responses often occurs -

"Sounds like you guys are hacking", or

"Agile uses strange new-age, communal development practices", or

"Agile only works for small, open-source efforts, not bigger projects", or

"Agile doesn't work when you're told the date you have to ship to customers (or clients)"

Basically, you can sum up the reactions to that of skepticism. Agile seems like something that is too good to be true - you can abandon the heavy structure and process found in most development efforts for a highly iterative, reactive model. People schooled in formal process/project methods would call that hacking, no doubt.

But there lies the first misconception about Agile and the various approaches to Agile that have developed around the world. The first thing to understand is that there is no one way to do Agile software development. There are several different mainline approaches and undoubtably many hybrids that are in use in the field. The one that many developers first learn about from magazine articles, hearsay and so on is the method popularized by Kent Beck called XP or Extreme Programming.

Beck describes XP as the following from his book Extreme Programming Explained:

XP is a lightweight, efficient, low-risk, flexible, predictable, scientific, and fun way to develop software

Later he goes on to state that:

XP is a discipline of software development

Note the use of the phrase "software development" as a way of placing XP's emphasis on the development part of a project. Although XP has mechanisms for dealing with how features are selected for implementation, and for estimation of those features, it focuses most of its attention on how developers can collaborate to develop those features more productively and with higher quality by introducing 5 principles:

• Rapid feedback


• Simplicity


• Incremental change


• Embracing change (i.e. change is a good thing)


• Quality

Rapid feedback stresses getting responses from the intended users of software as quickly as possible. This feedback is both to assess the features that were developed, and to guide what features will be developed next. This replaces the mentality of assuming that all features can be determined at the outset if only enough effort, people or time is expended.

Simplicity stresses building the simplest thing that could possibly work. That is, building only what is absolutely necessary to fulfull the expectations of the users. Many if not most software development efforts struggle constantly with how much effort to expend on architecture, infrastructure to handle assumed future features, new technology, flexibility, customization, and so on. Basically, developers are smart people and they like to show it by creating an application that can readily adapt to changing needs. They also like to work with new technology and use the latest architectural/design approaches. Left uncontrolled, this can quickly spiral into a system that is overly complicated and tragically is never as flexible in the end as was initially assumed.

Incremental change stresses making changes to a system bit-by-bit rather than the proverbial "big-bang". As features are suggested by customers, and changes are identified by the developers, these changes are introduced as time proceeds rather than all at once which would likely destabilize the system in the process. This value is one that would likely annoy developers particularly those that see the end design, architecture or whatever if only they could scrap the existing system (or part of it) and just.. rewrite it!

Embracing change stresses the concept of the developers assuming that change is inevitable in any system that is growing. A system that does not grow is either dead or dying. However, grow does not mean that it is only adding ever new features on top of existing ones. It also means it is modifying existing features for changing needs, and pruning ones that no longer are needed. In essence, it's the realization that a software system is never "done". Instead it is in a constant state of flux as it reacts to changing demands on it by users. This changes the time horizon for developers by focusing less on the destination and more on the journey.

Quality stresses the need for a system to be of high (or release) quality at all times. In a typical system, quality passes through stages where early on in its development, quality is very low as the system is constructed. Later as more of it is "finished" quality increases as features work as intended and defects are gradually identified and removed. Finally, at some future point (like just before the scheduled ship date), the quality is at an acceptable level and is ready for use by users. Gee. If it worked that well, we'd have no need for any other process. Too bad we know that this is almost never the case. Projects routinely spin off into the 95% complete level for 2 years because bugs are found at the rate they are fixed, or a project runs off a cliff with the bug rate increasing faster and faster even as bugs are fixed. I've been very fortunate to never have worked on a small or large project that suffered the fate of the latter, but I have seen projects in my midst that have. It's not a pretty sight.

Next time I'll talk more about the ways that XP provides techniques to support these principles.

Homework: Read Extreme Programming Explained by Kent Beck

Technorati Profile

Posted by Mark M. Baker at 10:14 AM  Permalink |

I've been away for a bit shipping some software, but I'm glad to be back blogging about .NET and the Windows world. Hope you all had a nice set of holidays, and haven't given up (yet) on those new year's resolutions.

Lots been going on in my software world since I last penned on Zen of Settings (which we're going to pick up again, I promise). The biggest thing is an immersion into Agile development using Scrum. I'll be writing more on this topic and how it intersects with .NET development as the year progresses. But so far, I'm really, really excited about how Scrum is working within my teams.

This morning in fact, I read a note from Ken Schwaber (the co-inventor of Scrum) about the Agile 2007 conference coming to Washington D.C. Definitely made my day since I live in the D.C. area. You'd think the capital of the U.S. would get a huge share of East Coast conferences but for some odd reason - to me at least - D.C. gets overlooked in favor of Boston or Florida. Peculiar since we're about 1/2 way up the coast and easy for people to get to compared to the other locales.

If you're anywhere near D.C. in August, I definitely recommend attending the Agile conference. Even if you're not doing anything with Agile yet, it's likely you will be at some point in your career.

http://www.agile2007.org

Posted by Mark M. Baker at 12:46 PM  Permalink |