Practicing .NET

Improving developer productivity and software quality

February 2007

Last time I talked about the Agile/Scrum notion of a Burndown. If you haven't read that entry yet, go back and read it top to bottom or the remainder of this entry won't have any perspective for you. At the end of that entry, I mentioned that sometimes a Burndown doesn't happen fast enough; that is, the work that proceeds through the month of the Sprint doesn't go to zero remaining hours (i.e. we're done) quickly enough. Scrum doesn't prevent this from happening, but it does give you tools to manage it, and adapt future Sprints to how much work can get done in a Sprint.

And this is the basis for any good (and realistic) schedule.

Unlike the formal, waterfall-style project management methodologies, Scrum doesn't focus much attention on projecting a schedule based on assumptions of time utilization, productivity, interruptions, and so on. Scrum says that this is basically impossible and is a reason why the vast, vast majority of software projects come in late or not at all. Essentially, in a formal approach you are trying to itemize every conceivable thing that could interrupt the work flow, weeks or months in advance, and setting an end date based on those assumptions.

So what happens if during a given work month, the team runs into problems with some 3rd party technology that seemed to work so well in the demo's? Or the team gets bogged down in the design of the user interface? Or team members get sick from that nasty virus making its way around the company? And on, and on, and on.

Answer: your schedule slips. Actually, the reality from Scrum's perspective is that your gloriously detailed schedule is probably slipping as soon as you hit Save and printed out a copy to hang on the wall both from anticipated issues that take longer to resolve than expected, and the unanticipated issues that you completely overlooked (gee, that's what a buffer is for, right?)

For most teams, they either work harder, longer or faster, or start cutting corners to make the promised date. In Scrum, this doesn't happen. Let me state that again, in Scrum this doesn't happen.

Huh? Does Scrum have a magic elixir to creating a perfect schedule? Actually, no. Scrum eschews the whole notion of a perfect schedule in favor of a work unit (ie. the time-boxed Sprint) that is organized on the principle of delivering highest business value features first. Scrum says that basically "stuff happens" to re-use a well-worn euphimism. That no one, and I mean no one is smart enough in a software project of any reasonable size to know everything that will ever happen and the time it will take before the software can ship.

More importantly, Scrum says that even if you think you know, you can't because what is knowable at the beginning may not be the same set of facts later on in the project when people change their minds, new priorities emerge, and so on. Strikes me as almost a Quantum Theory of Software Development. Hmm. Where's the cat..

Back to the Sprint now. So, the team is working along on the next set of prioritized business value items from the Product Owner during the month. As they proceed, they continuously update their Sprint Backlog (their ToDo list) and the Remaining Hours of work for each item. In Scrum, we focus on Remaining Hours, not Used Hours, Original Hours, or any other comparison to the past. We only care about how much work is actually left. The total of all of this work is summed per day. This data can then be graphed showing the relationship of work remaining to time. In the ideal world, the line would be perfectly linear. But in the real world, the trend line moves up and down as work is discovered, estimates are changed, work closes more quickly than expected, and so on.

But sometimes a Team finds that they overshot the mark when they agreed to select a certain number of Product Backlog Items during the Planning Meeting. Essentially, they bit off more than they could chew. Even in Scrum, this can happen. It's particularly common when a Team gets started in Scrum for the first time; they have no idea in reality how much work they can get done and often select way more work than they can do.

As the Sprint enters the 3rd week of work, the trendline of the Burndown is likely showing how things will end for the Sprint. If the trendline is not promising (ie. they won't make it), the Team gets with the Product Owner and notifies them that one or more work items won't get done in time for the Sprint. For many teams, and many Product Owners, this is the first real introduction to Scrum and Agile in general. Up to now, it's been nice theory and everyone feels good about the communication aspects. But now we get to see what happens when things don't go so well.

First, the Product Owner is likely to be an unhappy camper. But, that's reality. Yes, the Team could work weekends, nights and exhaust themselves to make the date but in Scrum's way of doing things, this is a very bad thing except in dire situations (like the company is going out of business if we miss the date). The correct approach is an honest discussion between the Team, Product Owner, and Scrum Master about the situation. Often, the work items can be simply moved to the next Sprint. However, if a work item is partially completed, the Team doesn't get to state this at the end of the Sprint; it's either done, or not done. There's no middle ground in Scrum.

Second, the impact of this is that the Team is forced to see that their capacity to do work is less than they thought using their current development practices. During the next planning meeting this information will be used to gauge how much work should be selected again. The challenge for the Team is to select a lesser amount of work for the upcoming Sprint based on this knowledge. In Scrum, this is called the Team's Velocityand is a way of describing the bandwidth of the Team. This is key to predicting how much time it will take to complete the feature set, enhancement list, etc that is expected by the Product Owner.

Next time, I'll talk a bit more about Velocity and why it's so important to Release Planning.

Posted by Mark M. Baker at 03:09 PM  Permalink |

In a classic waterfall (ie. death march) project, burning down would be that last 20% of the project where 80% of the work gets done. You know. Getting in at 6am, gulping lunch down, staying through the evening with cold pizza as a friend, gulping down caffeine and watching the days roll by all too quickly as the deadline approaches. The team, if they're lucky, makes it barely across the finish line usually by jettisoning things like a life boat with too much on board. Things like quality, features, documentation, anything to make the date. Sometimes it's enough, often it's not.

Been there?

No matter who I talk to regardless of whether they work for a large or small company, use the latest technology or don't, have genius developers or not, everyone seems to suffer this fate in software development. It's been studied exhaustively for decades and people have worked to come up with ever more creative ways to keep things on track. Better requirements, better and more detailed schedules, more tracking, more meetings, more "one more for the Gipper" sessions, and so on. Yet, the problem continues.

Why?

Basically, for a very simple reason. It's really, really, really hard to know everything that needs to be done on a project that takes more than a day or so to do. A day. Not a few months, not even a few weeks. Teams that think that they can, actually rarely do and only then by likely having done the same task before. And this gets to why software is so different from other work - it's always changing and is highly subjective. Designing an entry form for data could be a simple exercise or take weeks of implementation depending on the depth of usability that is desired, the platforms the application must support, the richness of the tools that are used, and so on. So many variables to consider. It's a wonder we've gotten software out at all.

Yet, an Agile project faces this same problem as any other. Scrum's solution to this is the burndown. That is, a way to see what is going on in the project and using that information to do two things - create an awareness within the team of how they are doing on the backlog items they are working on, and secondly, providing information about how much work, or as Scrum refers to it, as Velocity, the team can get done in a given unit of time (the Sprint).

As the team proceeds through the days of the sprint, they spend considerable time thinking, talking and adjusting their assumptions of what work they have left to do for the Sprint. These assumptions are what is detailed in the Sprint Backlog - that is, the set of work that needs to be done to complete the items in the Product Backlog - at any given day. That latter part is critical to understand. Scrum is always looking forward during the Sprint. It's always asking the team "what is left to do", "how much time is remaining for each task that remains", and so on. It does not look backward at how much time has been spent on a work item, whether all of the work items were discovered up front during planning, worry over metrics of estimates vs actuals, and so on.

You need to really, really think deeply about that for a bit. Once you spend time pondering this idea, you get a kind of Eureka! moment with Scrum. The "I get it" moment.

When a team asks me about the Sprint Backlog, they are often hesitant about what they are "allowed" to do to it. My response is the following - the Sprint Backlog is yours. You can do whatever you want with it. The tasks can change at a moment's notice. You can change your remaining hours whenever you wish. You can add, delete, modify and so on anything you wish. And you don't need permission for any of this. Your only requirements are that the Sprint Backlog reflect the team's best guess at remaining work for the Sprint each day, and each work item has a best guess at remaining hours before that item is done.

I usually throw out an example. Say you estimate a work item during the planning session as taking 5 hours. Maybe it's creating a basic data entry form based on the Product Backlog item. You start to work on Monday morning. During the day things don't go well. The tool you selected has issues. You spend the day on the phone with the vendor. You make some progress, but by the time you leave, you realize that this form is going to be harder than you thought. You stare at the remaining hours for a bit and decide that given what you now know, you need 15 hours to finish this task. Think about that for a minute. You thought you needed 5 hours the day before. You spent, say, 8 hours on it. You now think you need 15 hours to finish it.

That's just fine in Scrum. In fact, it's healthy. Because this kind of thing happens all of the time. One of the big ways that I think Scrum can fail for a team is if they start playing games with the tasks and remaining hours. You know, a kind of on-the-books, off-the-books set to keep the "boss" happy. Scrum says, "no way". The Sprint Backlog needs to be as close to what the team thinks is reality on any given day as possible.

And here's why.

As the days proceed, there is a running total of remaining hours of work in the Sprint Backlog. That is, you sum the the hours for each open task and you get a total. Ideally, this total goes down each day through the Sprint until it reaches zero on the last day. Nice linear line.

Never happens.

Actually, I should have said that "stuff" happens. As I mentioned earlier, people discover new work for an existing item, they discover new work items, old work items are closed or deleted, and so on. The sum of remaining hours per day fluctuates and can go up on any given day if new work is discovered. In a classic process model this would be a bad thing and cause meetings to ensue. In Scrum this is just noise in the system. It's tracked and the team monitors it. But work continues.

The trend of remaining hours is called the burndown.

But you probably already guessed that by now.

Next time, I'll talk about what happens if the burndown doesn't burn down fast enough.

Technorati Profile

Posted by Mark M. Baker at 05:14 PM  Permalink |

That title almost sounds like a hip little coffee shop here in downtown D.C. Alas, it doesn't focus on caffeine delivery but rather on how project teams meet at least once daily to discuss together their status. Yes, status. That feared word in every team (or office) of the endless "go around the table and talk about what's going on" meetings that bore people to tears and take 3x longer than they should. It's not a total loss if someone bothers to bring some donuts like those from our local sweet spot named the Fractured Prune; I especially like those little Strawberry Shortcake ones. But I digress.

Actually, one of the few, but key parts of Scrum is its focus on the Daily Scrum (or daily meeting). Unlike a typical meeting, a daily scrum is time boxed to 15 minutes regardless of team size. Imagine that - a meeting that lasts a maximum of 15 minutes. In practice, mine last around 10 and we zip around and get through business pretty quick. And it's a stand-up affair. No lounging back in some comfy conference chair dozing off until it's your turn.

Here's how it works.

The team, and I mean the whole team, meets at a designated time each day during the sprint. Typically the meeting time is beginning of day. In my case, we have one at 11:45am; it's just before lunch and people are motivated to keep it short. In addition, my team is geographically dispersed along the East Coast and in Canada. So we have to do this each day via conference call and with team members that are here in D.C. But it works well despite the logistics.

Each team member is expected to discuss 3 and only 3 things each day:

What the person did yesterday (ie what they accomplished)

What the person expects to do today (ie their goals)

What impediments/blockages they face (ie what's holding them up)

The format is a part of the Daily Scrum format and is followed rigorously. If team members start having a "conversation" around an item, the Scrum Master gently clips the conversation and suggests it be held afterwards. If team members are matrixed such as DBA's, Network Services, etc they may have done non-Sprint related work the day before. Those items are not discussed as they are irrelevant to the Sprint itself. Impediment discussion is a way of alerting the team and the Scrum Master to upcoming or existing blockages that are affecting forward momentum. It's up to the Scrum Master to work to see that these are eliminated.

Sounds straightforward and easy compared to doing major prep for a standard status meeting. But it's deceptively simple.

When the team goes around the "table" talking about their 3 items, they address each other, not the Scrum Master, not the Product Owner, or anyone else perceived as the "boss". Sometimes in teams this can happen if the Scrum Master is perceived as the boss; the team will address their comments to the boss rather than to each other and this defeats the team nature of Scrum. In my case, I'm the engineering manager as well as the Scrum Master. But I made it a point to drill into the entire team that my role is to be the Scrum Master-only in these sessions. So I walk around, stare at my notepad, and just listen. With encouragement, they talk directly to each other or into the phone as in our distributed team arrangement.

The idea of the Daily Scrum is that it aids in the essence of what Scrum actually is - a giant feedback loop. In order to be a feedback loop, information must be acquired constantly and re-inserted into the system so the system can recognize the data and make adjustments. The Daily Scrum ensures that everyone is focused on progress being made, what work is being started and what is keeping them from accomplishing the goals of the Sprint. Without this constant, daily discussion, the team may assume that things are going well, when in fact they aren't.

In practice, I've found that a cross-functional team of people gets to the point of looking forward to these daily, very short meetings. It cuts down on surprises and ensures that people know what is going on at all times. It also helps in tracking forward progress in the Sprint using something called the Sprint Burndown Chart.

Which I'll get to next time.

Technorati Profile

Posted by Mark M. Baker at 04:06 PM  Permalink |