The current software development situation is less than ideal. Our systems are regularly delivered late or over budget—if they're delivered at all. Our systems often don't meet the needs of our customers, and must be redeveloped again and again and again. Our customers are angry with us because of these problems, and aren't willing to trust us or work with us again because they've been burned too many times in the past. To make matters worse, our customers don't really understand what we do, how we do it, or why we do it—resulting in unrealistic demands and a lack of the support that we need to accomplish their goals.
It isn't much fun for software developers, either. We work overtime—way over—typically 50, 60 or 70 hours a week, and quickly become burned out. When projects run into trouble, often before they've even started, we point fingers at others. Convenient targets include our "pointy-haired bosses" whom we believe are barely competent enough to tie their own shoes, the "paper-pushing fools" in the department down the hall who demand excessive amounts of documentation, and our "stupid users" who don't know what they want (and when they do tell us, their requests never make any sense anyway). Naturally, we never blame ourselves—after all, we're perfect. So what do we do when we realize that a project is in trouble? Sometimes we give it our all, working excessive hours in a futile attempt to meet unrealistic demands, embarking on a death march (for more information, read Ed Yourdon's article, "Surviving a Death March Project," July 1997, and his book, Death March: The Complete Software Developer's Guide to Surviving "Mission Impossible" Projects [Prentice Hall, 1997]). At other times, we disconnect from the project entirely: Realizing that it's doomed, we decide that we should at least learn something useful to pad our résumés, so we download some new development tools from the Internet and start playing with them.
Rethinking Our Approach
How have we gotten ourselves into this mess? First, many people have lost sight
of the fact that the primary goal of software development is to build systems
that meet the needs of their users in the most effective manner possible. For
the sake of this discussion, a system includes software, documentation, hardware,
middleware, installation procedures and operational procedures. I suspect that
this oversight has occurred because the development philosophies and approaches
of one, if not two generations of IT professionals have been shaped by prescriptive
processes and the gurus promoting them. Prescriptive software processes (a.k.a.
heavyweights) such as the Unified Process and the OPEN Process have their place,
but they may not be as appropriate to every situation as their supporters claim.
Mere Cogs in the Wheel?
Heavyweight approach processes typically focus on prescriptive procedures and
the artifacts that should be created—approaches that are often implemented
by organizations who consider people to be "plug-and-play–compatible."
In other words, they believe that with the right process in place and the necessary
amount of artifacts created, you can swap people in and out of a project with
relative ease. This is true only when the person you're replacing isn't very
productive—which is often the case in organizations following a heavyweight
process. In reality, replacing a productive person is problematic, regardless
of the process being followed, so the "plug-and-play compatibility"
approach is questionable at best.
Though heavyweight processes are attractive to management, most developers don't share that enthusiasm. Heavyweight processes are typically based on a command-and-control paradigm that puts management in control—well, at least makes them think that they're in control. When the going gets tough, though, developers quickly abandon the process—unfortunately throwing out the good with the bad when they do so—and often find themselves in an even bigger mess than before. These desperate shortcuts often lead not to salvation, but to quagmires, turning your road to glory into a death march.
It's Our Fault, Too
The way that developers learn their trade has a few unique dysfunctions. For
the most part, our colleges and universities do a reasonable job of educating
developers for entry-level jobs. Note the use of the word reasonable. Not only
would I like to see schools produce technologically competent developers, I'd
also like to see students gain "softer" communication skills so they
can interact more effectively with their coworkers and customers. It's clear
that schools need to place a greater focus on teamwork, testing, usability and
software modeling.
Ah, Youth!
However, even if schools were doing a perfect job and everyone was getting a
degree or diploma, we'd still have a problem because of the inherent nature
of software developers. When software developers are in their teens or early
twenties, their priorities are typically focused on learning and working with
the latest technology. They describe themselves as PERL hackers, Linux experts,
Enterprise JavaBeans developers or .NET programmers. But because the technology
is constantly changing, younger developers tend to just barely learn a technology,
use it on one or two projects, and then start over again, either learning something
completely new or absorbing the latest incarnation of what they worked with
previously. The problem is that they keep learning slightly different flavors
of the same low-level, fundamental skills over and over again, a problem that
many developers become aware of only after several rounds of technologies. Once
you've written code for transaction control in Cobol, Java and C#, you begin
to realize that the fundamentals don't change. The same can be said of database
access in various environments, user interface design and so on. Before long,
developers start to understand that many of the fundamentals, which they may
or may not have been taught in school, remain the same regardless of the technology
employed.
The Circle of Life
By the time developers begin to realize this, they're often in their late twenties
or early thirties—typically the age when people start to settle down, get
married and buy a house. This is fortuitous because these new demands mean that
developers can no longer afford to invest vast amounts of time learning new
technologies; instead, they want to spend that time with their families.
Suddenly, higher-level roles such as project lead, project manager and modeler become attractive because they don't require the intensive effort involved in learning new technologies. So, by the time that developers begin to truly learn their craft, they're transitioning out of their roles as developers. Fortunately, new "young punks" come along, and the cycle repeats itself. Ironically, this cycle ensures that the majority of those actively developing software are not the best ones qualified to do so—and worse, don't even know it.
Watts Humphrey, founder of Carnegie Mellon University's Software Engineering Institute (SEI), observed that many developers with 10 years of experience actually have one year of experience 10 times over. Perhaps if he considered developers with 20 years' experience, he'd discover people with 11 years of experience plus another year of experience nine times over.
Our "Challenged" Customers
Things aren't much better on the business side. Our customers don't understand
how software is developed, which is reasonable because that's not their job
to begin with. Can you tell me how a marketing campaign or a human resources
strategy is developed? Likely not, so fair is fair. Furthermore, our customers
generally aren't all that interested in software development processes; they'd
rather leave the details to us so they can get back to doing their own jobs.
They're often happy to be involved in a requirements workshop or two at the
beginning of the project, to review key documents throughout the project, to
receive glowing (albeit sometimes falsified) status reports throughout, to get
involved with acceptance testing just before delivery, and finally to receive
the system eventually (often late and over budget). This is the way that it's
always been, this is the way the IT professionals (following older processes)
tell them it has to be, and they often don't question the status quo. It is
a mite strange that they tolerate this situation: what they're asking for is
software that meets their needs in an effective manner, but what the developers
are giving them is a bunch of documents to review, some status reports, some
tests and finally, some software—if things go well. In other words, current
practice today is to deliver what we want to deliver, not what is asked of us.
Just the Facts, Ma'am
The first step in addressing a problem is to recognize that you have one—and
we definitely have a problem. Our approach to how we organize ourselves, both
at the department level as well as at the project level, must reflect the facts.
The fact that novice developers will require years of experience to gain a grasp
on the fundamentals of their job shows me that organizations need to pair these
green sprigs with grizzled veterans to facilitate an apprentice/ mentor relationship.
The fact that prescriptive processes are often abandoned by developers indicates
that organizations should seek a more palatable approach that still works; perhaps
an agile methodology along the lines promoted by the Agile Alliance (www.agilealliance.org)
and my own Agile Modeling (www.agilemodeling.com).
And the fact that our users don't understand how software is developed tells
me we need to invest the time to educate them about what we do and why we do
it, so that they can work with us effectively. Yes, software projects can succeed—it
happens all the time. The challenge is to ensure that it happens more often.