Our story takes place at "E-Co," where I was dropped in as the acting Chief Technology Officer. E-Co classifies itself as a "new media company" in the coming media convergence revolution of television and the Internet. It stages media spectaculars, which it then webcasts and uses to market sponsors' products.
Aside from the incomparable quality of our beer, we Canadians love to harp about how our country is the best place in the world to live. For years, according to the United Nations Human Development Index, which measures quality of life by factors such as education and life expectancy as well as per capita income, Canada ranked number one. This gave many Canadians license to chide our American friends about their country's poor showing (merely number 12). UN officials repeatedly scolded Canadians because the index was never intended to be misused in this manner—it was designed to classify countries for development programs, not for propaganda purposes. (By the way, you don't hear a lot about the index around here anymore after Canada slipped to third place this year behind Norway and Australia.)
Just as countries can sometimes misuse classification models to lay claim to a superior quality of life, the software industry misuses classification models like the CMM to claim superior product quality.
Classifying Maturity
The Software Engineering Institute's (SEI) Capability Maturity Model (CMM) is
a five-step model for ascertaining the maturity of software development organizations.
This score is determined with a survey of the company's compliance with the
CMM's Key Practice Areas (KPAs). KPAs are what I would normally think of as
industry best practices, including version control and defect tracking. The
majority of software development organizations are level 1, which is considered
random or undisciplined processes. Companies that follow minimal best practices
such as defect tracking and version control move up a step to level 2. Institutionalize
these practices, and you climb another step up the maturity ladder.
The CMM was originally developed to provide the government an objective scale to evaluate its contractors' software development maturity. It has become the means by which many in the software engineering field rate the maturity of their software development organization. It's a good model for evaluating your software engineering capabilities and identifying areas that you can improve on, but, just like the UN Human Development Index, it can be misused for political and commercial purposes.
Case in point: A while ago, I was talking to the development manager of a company that creates human resources management software, discussing the bids that I had received from different vendors for our e-commerce system, some of them in the million-plus region. He stopped short of telling me that I was insane to outsource to North American software houses, but enthused about his company's experience in outsourcing to India. For just 40 bucks an hour, he said, I'd have access to top-notch programmers.
I remembered that, in university, a lot of my friends were visa students from India. Many stayed in Canada, but quite a few returned home. They were good developers, and if I could hire high-quality talent for $40 an hour instead of the $175 an hour I was being quoted by most development houses—well, my friend was right: I'd be crazy to outsource to a North American software house.
My colleague introduced me to their outsource vendor's agent, a pleasant, enthusiastic chap who gushed about his company's technological prowess: leading-edge software technology, object oriented, RUP, Use Cases, UML, Rational Rose. And, about every fourth sentence, he was careful to mention that they were CMM level 5: To maintain their certification each year, they had to fill out the CMM survey and identify their implementation of the CMM KPAs to the auditor's satisfaction.
Though I view certification as a good sign, I'm always a little skeptical of a company that considers certification a guarantee of quality. A few years ago, I participated in a local corporate consortium about ISO 9000, another certification for process quality. Our favorite joke was "You could make lead life jackets and still be ISO 9000 certified." Also, I'm a little cautious about certification processes in general—friends working in the defense industry have told me too many stories of being "coached" before filling out various quality audits.
My friend showed me a sample of the company's work products, starting with
use cases. They were all CRUD-type use cases, and I could classify them as either
Create X, Read X, Update X or Delete X—use cases in name only. Though they
may have satisfied the CMM requirements, they certainly didn't delineate the
value of the system to its users. The class model was a poor combination of
massive "God" classes and data classes, with nothing more than set/get
operations for every attribute. There was nothing object oriented about the
architecture—a CASE tool like Rational Rose isn't a magic OO wand, and
automatic code generation from a bad model does not create great code. The use
of such tools may be a sign of software maturity, but it's no guarantee of product
quality.
Caveat Emptor
Despite my concerns, my friend was happy with the vendor because he was receiving
the benefits of a mature software development process with regular, predictable
delivery of product at a defined level of quality—at a bargain cost. Unfortunately,
sometimes you get what you pay for. With minimal software experience, my friend
wasn't aware of the product's brittle architecture, nor did he anticipate the
high cost of inevitable feature changes and enhancements in the future. As his
product was designed to serve as the framework for a family of related products,
this was a real disaster in the making.
With E-Co, I'd flirted with disaster long enough. When I told the vendor agent that we wouldn't pursue business with them, his entire counterargument was, "But we're CMM level 5!"
Maturity Mantra
Well and good—but I wanted more. Karl Wiegers wrote: "The CMM assumes
that God gave you the requirements and you take them from there." My requirements
specifications were anything but divinely inspired. I needed a vendor who would
work with me to quickly resolve requirements issues, with the ability to recommend
and implement improvements. The vendor's mantra "We're CMM level 5!"
made me wonder if that certification would be used as a shield to deflect any
complaints, framing product defects as E-Co's responsibility because of our
imperfect requirements. I simply wasn't confident that this vendor had the ability—or
desire—to protect E-Co's interests.
While I strongly believe in a disciplined software development process, a high CMM level does not necessarily imply high-quality products. Great configuration management means that you'll always know what went into a build, but it won't improve the flexibility of the software in that build. Great defect tracking means that you probably won't ship software with the same defect twice, but it won't guarantee a great solution for that defect. To produce high-quality software, you still need a high-quality software development team.
Moral: There are lies, damn lies and maturity models. Beware of attempts to use the CMM beyond a statement of software development process maturity.
Next: First Flight
