It was a dark and stormy night ... election night. The wind and the political commentators were pouncing like black widows on every bit of precinct data as it was released, eager to spin a tale of victory or defeat around the candidates for public office.
I sat at home swollen with pride and confidence as I prepared to monitor the county website application I'd developed to publish results as precincts reported. The data was mirrored straight from the SQL Server database the Board of Elections used, so it was as near real-time as was practical, refreshing every five minutes. I'd set up automated e-mail notifications that reported results to various members of the press and candidates every 15 minutes.
I still remembered the painful first election after the system was implemented. Our CIO had proudly announced it to the press, who gave it plenty of pre-election hype. That night, I learned more about the impact of limited bandwidth and the importance of caching than ever before. At one point a local television commentator said, "If anyone from the county is watching, please call us and let us know why response time is so slow. Oh, the humiliation.
After that election night debacle, we quickly upgraded the server and modified the program. Instead of dynamic ASP queries, we used SQL Server's ability to publish static HTML pages, which would cache and net better response time for the eager masses. Smiling, I watched as the election results started to trickle in, focusing on the pivotal sheriff's race. The favored candidate was clearly in the lead.
Cardiac Arrest
Minutes later, I refreshed my screen, and a cold chill crawled up my spine. Results that had been posted there had disappeared! The popular candidate was down to negligible votes, and my ears were ringing with tomorrow's headlines, as I imagined them: Incompetent programmer foils election, costing taxpayers thousands.
Frantic, my heart palpitating, I scoured my stored procedures and views, wondering where I had gone wrong. My desperate calls to the election office rang unheeded—after all, they were dealing with the enraged masses crying foul, clamoring for answers. My panic attack worsened when I thought about the e-mails being fired off to every newspaper in the region, reporting these questionable-looking results.
I eventually learned that my program was happily working just as it should—and that the Elections office had retracted votes in order to verify them. Development horror stories are fun to tell after the crisis is over, hovering over a hot computer on a cold night. But floundering in the bowels of the experience is a palm-drenching, stomach-lurching nightmare.
—Donna Davis
He Who Must Not Be Named
This program's cynical moniker turned out to be a self-fulfilling prophecy.
A few years ago, I was working on a critical billing system used in a real-time environment. Of course, to make life much more fun, the system was in its end-of-life phase and a brand-new system was supposed to arrive at any time. The oldie was built on the jolly good client/server model, with loads of unreadable stored procedures and a fat client that would have given a Swing client reasons to boast about its figure.
For some reason, the system was named after a mythical beast that I will not disclose here (but Tom Cruise could fill you in on it). I soon discovered that only sheer premonition could have made the developers christen it with the name of a monster.
The stress on the system was maximal, as it was on the team: The slightest disruption of the ongoing process could quickly accumulate huge backlogs of data that had to be manually fed back into the system after the issue was solved—that is, on nights and weekends. This caused some billing items to be pushed to the next invoice, and sometimes even to the following one. We could imagine customers getting exhilarated about their tiny bills only to get socked silly one or two months later. With direct debit from a checking account, this was usually enough to elicit the infamous phone call from the banker ("Er ... we have a problem ...).
I thought that ominous phone calls from the bank were the gnarliest aspect of this insidious program, but I was wrong. Twice, during lunch conversations about the monster system, we had unexpected encounters with irate customers who happened to be dining at a nearby table—and eavesdropping. They were furious that we were the culprits behind the billing process that was causing them such grief, and they made their displeasure audible to the entire restaurant. Interestingly, people tend to lose all sense of humor when it comes to the bottom line. And yes, when faced with irate, cutlery-wielding victims of your poorly designed system, you can see your whole life come unstrung in a millisecond.
There's a moral to this story. You should never, ever give a monster's name to a program: The thing could awake, break free and make your world a much more dangerous place.
—David Dossot
Cofounder, V.P. Technology
Agile Partner,
Luxembourg
From Vampires to Champagne
Oh, for agility in the days of old!
One project always returns in my nightmares. I'm the systems engineering manager, wild-eyed and desperate, holding up a silver cross in a futile attempt to fend off a band of blood-sucking marketers with gore dripping from their gaping maws. As I back away, a wild group of hungry, yelping engineers on their hands and knees surrounds me, snapping at my heels, demanding their dinner. I wake up in a sweat, heart pounding, gasping, and oh so relieved that it's just a bad dream—now.
But at one time, it was cold, cruel reality. It wasn't pleasant being a gatekeeper then. The company's systems engineering group had created a new division to develop products outside the umbrella of its main businesses, to transition from the mechanical to electro-mechanical age. Trouble was, although the engineering groups were new hires (including myself), the marketing group was part of the old guard.
My group was responsible for the overall architecture, specifications and project schedule, put together with input from both the marketing and engineering managers. Simplistically, requirements and delivery date came from marketing, systems engineering turned it into requirements and functional specifications, and with some back-and-forth between engineering and marketing, a project schedule was created. I had periodic meetings bringing together marketing and engineering to expedite communication. All changes and new functionality came through my group. In many cases, we'd initiate meetings in which features were reprioritized if the date was immutable. Marketing, of course, wanted it all—everything was equally important and the deadline couldn't be moved—and engineering, eager problem-solvers that they were, would say it was simple to implement. To make matters worse, the engineering director loved to jump in and offer the perfect solution that he could do himself—which, given his time constraints, he couldn't.
Eventually, marketing personnel would go casually to engineering to request a feature or enhancement, and engineering would say "Sure! The specifications grew more and more obsolete, and eventually the entire project schedule began to slip. Sound familiar? After a year of this frustration, I returned to R&D, where we could leap-frog to the electronic age, explore how to do things without time constraints, escape marketing and really have some fun. Most of the engineers left soon after I did (I hired some of them into my new group), and a year and a half later, the product was finally released—late, of course, ridden with bugs, and missing some key functionality that the customers had really wanted.
I wish I'd known about agile development techniques then. The teeth-grinding tension would have been alleviated by quickly delivering functionality for the customer, and prioritization would have been an integrated process, with all parties agreeing on what was to be developed first. Not all specifications for all features would have been completed up front, and a more fluid, cooperative mentality would have replaced the combative us-vs.-them philosophy.
And, of course, my dreams would be of champagne toasts, not monsters at my throat.
—Rosalyn Lum
The Wrong Business?
The alarm bell buzzed, the red light flashed, yet I still went ahead with the project. What was I thinking?
The online request for proposals had seemed perfect: A local company needed help with a short-term project in one of my specialty areas. Perfect not only because I was sure I could do the work, but because their deadline was two weeks before I'd planned my overseas vacation. The principals were two guys fronting for their Mumbai-based development team. The offshore team was charged with developing the front-end and database stuff but had no experience programming airline reservation systems, a field in which I've done quite a bit of work. But beneath the shiny veneer, alarm bells started ringing in my head: They were committed to deliver in six weeks, but hadn't even hired a person to develop the core functionality? On the other hand, I knew I could deliver quickly, and one of the things I like the most about Web services is that it's easy to enforce decoupling—both sides would stick to an industry-standard XML specification, and that would be that.
They asked how much I thought the project would cost. I told them my best hourly rate. They said they preferred a fixed bid, so I gave them my hourly rate for six weeks, which we'd agreed was the necessary time period. They countered with a price less than half that. The alarm bells in my head were now supplemented with flashing red lights, and I left as soon as possible.
When I got home, I sent them an e-mail intended to sever our relationship: "Having reviewed your needs, I realize that not only can I not deliver the project for your proposed amount, I couldn't even do it at my regular rate. I couldn't possibly tackle this project in this timeframe for less than 150 percent of the original amount I quoted you. They called me up and agreed. I did something I've never done before: asked for a good-faith deposit. I got it, I started coding, I delivered on time, everyone smiled and shook hands, and I sent them my final invoice.
Which they didn't pay. They just ... didn't ... pay. I called them up. I had ignored their phone calls, they said, delayed the project, and cost them at least the amount they were stiffing me. "When? I asked. They named the weeks I was out of the country. "You'd already accepted the work by then! I protested. They still didn't pay.
I called my lawyer, who called me back promptly after he spoke to their lawyer, so incredulous at their attitude that he offered me a discount if I chose to sue. "How much of a discount? I asked. He told me what it would cost. "Steve, I said, "If they ever start offshoring law, you're going to be in for a rough time. "Yeah, he chortled merrily, "but in the meantime, my car payments just keep coming due. "You're charging me for this call, aren't you, Steve? I asked. "You're in the wrong business, buddy, he laughed and hung up.
—Larry O'Brien
Software Development
Industry Analyst and Programmer
Kailua Kona, Hawaii
Isn't It Ironic?
As project manager for a failed release, I managed to save the bacon. My reward? I was dumped.
Have you seen that poster with the six stages of a project? It starts with "Unbounded Enthusiasm and ends with "Punishment of the Innocent and "Promotion of the Uninvolved. Well, I lived through such a trauma.
As is customary with most nightmares, it started out well. I was put in charge of a small team. We developed some cool new features for our legacy product, a sophisticated machine containing software, electronics and lasers, all encased in a 20-foot steel structure. We kept our customers happy, and life was good.
Meanwhile, another team was developing the next-generation product with new technology. Everything was new—the software, the electronics, the lasers and even the steel structure. Unfortunately, things weren't going well. To start with, there were no written specifications, no scope management and no documented interfaces. A few of the senior developers, including myself, tried to suggest some changes. They were rejected. So we watched as the new team floundered, receiving weekly notices boasting that they were "real close now. They were so confident that they decided to schedule the first delivery, despite their noted lack of success in the lab! To top it off, the first recipient would be one of our most difficult customers—and there was no contingency plan if things went wrong.
Soon after the shipment of an embarrassingly shaky product, our company went through a reorganization. The new manager didn't like the way the project was going—who would? So he appointed me as project manager. I suppose that was my reward for making too many suggestions. I did my best, but it wasn't pretty. I made a new schedule and promised a stable product in six months. I reorganized the team so that some could continue fighting fires and others could make progress on unfinished features. Working long hours, the team managed to overcome every challenge. Six months later, we had our stable product.
So how did the new manager reward us? I suggested that we take the team out for a celebratory dinner, but he was too busy for that. However, he did find time to replace me with a yes-man who knew nothing about software. It fits the pattern perfectly: punishment of the innocent and promotion of the uninvolved.
It's now six years later. I have long since left the company. The people who still work there say that morale is horrible. I wonder why?
—Hugh Bawtree
CEO
Bawtree Software Consulting
Salmon Arm, B.C.
