Email	Print Reprint
	add to:
	Del.icio.us Digg Google Furl	Slashdot Y! MyWeb Blink

Let me start with three basic assumptions. First, software is arguably the world's most important economy. While I acknowledge and accept that software neither feeds the hungry nor comforts the afflicted directly, it is undeniably the fuel that has driven global markets to their current levels of efficiency. It's the catalyst that has made new ways of doing business and connecting people possible. Thus, indirectly, its existence has changed the human condition.

Second, building interesting software-intensive systems is still terribly hard. Contemporary tools, languages and methods have made it possible for us to build simple systems easily and, for that matter, have lowered the bar as to what we mean by simple. In other words, what we may have considered complex a few years ago looks much simpler today.

However, market and technology pressures lead us to build systems of increasing complexity, and it becomes exponentially hard to master that complexity. Most interesting systems can't be built by a single person, and thus you have the added complexity of managing a team of developers. Additionally, most interesting systems are built upon intrinsically difficult technology—meaning complexity has been heaped upon complexity. Despite advances in contemporary tools, languages and methods, developing complex software systems will likely remain a very labor-intensive business.

Third, there are no imminent breakthroughs in software engineering that will radically ease the lives of software developers. Some things have offered incremental relief (for example, UML, RUP, the phenomenon of open source and the emergence of extreme programming). And there are other developments that will likely help over time as they mature, such as Web-based software engineering. But ultimately, there is an intrinsic intellectual complexity in software development that can't be overcome easily.

Reducing the Friction
In the presence of this complexity, the best we can do is reduce the friction in software development. Whereas the presence of software has reduced the friction in business interactions, we still have a long way to go to increase the efficiency of software development processes.

As such, the history of software engineering is one of growing levels of abstraction. This is simply because abstraction is the central means whereby the mind can attack complexity. Here are a few examples of what I mean:

• Programming languages have grown from the expression of algorithmic to object-oriented abstractions.
• The patterns movement has given us a way to express common solutions to common problems.
• Modeling languages such as UML have emerged to help us express even higher-level architectural concepts that transcend the language or pattern details.

I'm also encouraged by the maturation of the profession. There is an emerging body of knowledge on software engineering found at the online Guide to the Software Engineering Body of Knowledge (www.swebok.org). However, there are a number of natural forces in software engineering that aren't easily deflected, despite the growth in abstraction that may help us manage complexity intellectualy. These include cost, schedule and functionality; compatibility; performance, capacity and scalability; reliability, availability, security and fail safe and fault tolerance; and technology churn and resilience. These elements are shown in Figure 1.

Figure 1. Natural Forces in Software Engineering These forces weigh upon the software development team in varying degrees.

Cost, schedule and functionality. These are the most obvious forces that weigh upon the software development team.

Compatibility. This is a factor in the sense that the team must address issues of legacy (compatibility with the past) and platform (compatibility with technology standards beyond the developer's control).

Performance, capacity and scalability. These represent the forces of the most important nonfunctional requirements upon a system. Advances in hardware have helped many a development team mask sloppiness, but for a number of system classes, you simply can't hide from these forces. Missing performance requirements is a showstopper, especially for hard, real-time systems. For Web-centric systems, it's difficult to plan for performance and capacity. These elements tend to grab you at the most inopportune time, namely when you have a system that begins to fall in on itself like a house of cards as it becomes successful. In either case, more often than not, simple local changes are insufficient to shore up what is generally a limitation in the system's scalability.

Reliability, availability, security, and fail safe and fault tolerance. These represent the system's response to unwanted external forces (such as hardware failure or attacks by outside agents). These forces are particularly difficult to address, because their origin is typically outside the development team's control.

Technology churn and resilience. Technology churn is particularly onerous, because it represents infrastructure changes—such as changing operating systems and various standard protocols—that lie beyond the team's control. Resilience is problematic because it touches upon a key business decision: Should we build disposable software now, so we can meet our immediate schedule but then incur the high cost of future change, or should we build for change now?

Most interesting systems are continuously changing, meaning they consist of thousands of moving parts, some of which change at different rates than others over the life of the system. Unless well-architected, with consideration for a clear separation of concerns and a balanced distribution of responsibilities, such changes will literally tear a system apart.

Depending on your domain, your development culture and the specifics of your problem, you'll find the forces mentioned here weighing on your team in varying amounts. However, in each case, it's the development team's task to balance these forces to yield a reasonable solution. What distinguishes mature from immature development teams is their ability to do so in an efficient, predictable and sustainable manner.

Balancing the Forces
In the war of balancing these forces, resolution typically takes place on three battlegrounds: the conflict between schedule and quality, the battle between high- and low-ceremony, and the changing nature of the development team.

The conflict between schedule and quality. Compress the schedule, and quality generally suffers. The reverse is not necessarily true: Seeking high quality doesn't have to extend a project's schedule. Indeed, the cliché holds true: If you have time to do it over, you probably had time to do it right the first time. The problem is, however, that you probably didn't know enough the first time around to succeed.

Too often, a schedule is an inflexible parameter, forcing the team to compromise functionality, quality or both. In the heat of battle, schedule pressures will dominate, typically because delivery, rather than quality, is a more concretely measured business factor. Balancing schedule and quality requires active intervention by the development team, in terms of intentional testing and quality assurance plans, as well as focusing on architecture first, to drive technical risk to the surface earlier rather than later.

The battle between high- and low-ceremony. By high-ceremony, I mean the use of heavyweight processes that dictate the rhythms of individual developers as well as the team as a whole. The problem with this is it typically stifles creativity and leads to horribly over-engineered solutions.

At the opposite extreme, low-ceremony projects rely upon the presence of extraordinarily talented people. The problems with this are many: It's neither predictable nor repeatable, there are far fewer extraordinary developers around than the developers themselves would acknowledge, and delivering a solution involves more than just great programming.

In my experience, there is a sweet spot in the middle, slightly skewed to the higher-ceremony end, which the mature software development organization must achieve to balance the forces of software.

The changing nature of the development team. For simple applications, a good team of programmers is often all you need. For most interesting systems, that team will expand to include lots of nonprogrammers, from content creators to testers to network engineers and so on. This conflict is most acute in the world of Web-centric systems, where content creators and software developers generally come from two distinct development cultures and use different tools and processes, yet both are contributing to the same product.

Content creators will be familiar with tools such as Macromedia's Dreamweaver, Adobe's Photoshop, and content management systems from Vignette and Interwoven; software developers will be conversant in the intricacies of J2EE, Rational's ClearCase and so on. The challenge is simply that content creators need some significant software in the back end to drive an enterprise site, and software developers need significant content to provide substance to their applications.

Mitigating Conflict
What, then, do we do to mitigate these forces and conflicts? For many organizations, the answer is to, intentionally or not, write disposable code, meaning code that can be scrapped and reworked. Now, I don't mean that in a pejorative sense: Some applications warrant this approach, especially those whose parts change rapidly. However, for most systems, this is a terribly inefficient strategy.

Alternatively, the best approach to mitigate the risk in software development is to write less code. Now, that can manifest itself in a number of ways: the use of higher-order languages, modeling and component-based development.

The growth of abstraction in software engineering is nowhere more apparent than in the evolution of programming languages. From Fortran to C to C++ and now Java and C#, there has been a marked improvement in the expressiveness of languages. In particular, JavaBeans gives us a mechanism for componentizing our abstractions, and C# gives us a start at binding our languages to services.

As for modeling, UML has definitely entered the mainstream, and is being applied in places far beyond its initial goals. Today, we see bindings of UML to every major programming language, to EJB, to relational databases, to XML and beyond. As such, if you extrapolate UML's trajectory, it appears to be on the path of becoming a complete visual programming language.

On the one hand, I welcome that development: The semantics of UML transcend, yet are complemented by, traditional textual languages. On the other hand, there is the challenge of keeping UML from growing too large. In short, the challenge for the UML 2.0 effort will not be what to add, but what to take away and consolidate.

During the development of the original UML specification, you could almost hear the metamodel breathe. Some days, it would grow in size in complexity. Other days, it would shrink, like when we'd discover some commonality we'd previously overlooked. I knew we'd made a breakthrough whenever the metamodel became simpler. However, making complex things simple is hard work, and doing so will need to be the focus of UML 2.0.

As for component-based development, the value of building systems from components is obvious. Yet one must ask why there isn't a flourishing market for components. Granted, there are companies in this public space—ComponentSource, IntellectMarket and others—but in terms of capitalization, this market hasn't broken through yet. The reason, I believe, is two-fold.

First, the presence of the necessary technical infrastructure to support component-based development is only a recent advance (J2EE in particular gives us the means of componentizing yet larger things). Second, I expect that the greater movement will be in the private market for components. All things being equal, if a company invests in componentizing its assets, it will be far better off using those components to dominate a market rather than trying to sell the components to others. Good components are a competitive advantage to an organization, and there is no amount of money that can be put on that value.

It's the emergence of the underlying technical infrastructure that has lead to the renaissance in scripting language. As has been said by others, Perl is the duct tape of software, enabling developers to lash together disparate parts quickly. Additionally, the emergence of standard protocols, from XML to WebDav (Web-based Distributed Authoring and Versioning) and beyond, has made it easier for part builders to specify their components and for part assemblers to put them together.

As the component market continues to mature, I expect we'll see a subtle shift therein, somewhat away from the componentization of bags of bits, to the componentization of services. That is what Microsoft's dot-Net (.Net) is mainly about, and Sun and IBM are making moves in the same direction. Why this shift? First, service infrastructure is fundamentally hard, so you want to build upon existing services rather than start from scratch. Second, on the Web, there is a transparency of space that, over time, favors the use of services. Indeed, this is much of the logic behind the emerging ASP (Application Service Provider) market.

May the Forces Be with You
As I said earlier, our industry has a long way to go to improve the efficiency of software development. Modeling helps, components help, well-defined processes help, but none of these alone represent breakthroughs that will yield a state change in the way we build interesting systems. Development is a team sport, especially in this Web-centric world, and it requires all these things and more for us to create the illusion of simplicity in this complex world.

RELATED ARTICLES This Week's Developer Reading List This Week's Multicore Reading List A Structured, Unified Approach to Multi-Core and Many-Core Computing -- with Applications Logging In C++ : Part 2 OutSystems Updates Agile Platform		TOP 5 ARTICLES No Top Articles.