Celebrating Failure: The Invisible DBA and Other Stories

Every year, there are more mishaps to report. Take the Minneapolis HMO that authorized free drugs in 2004 for nearly 30,000 retirees due to a programming error, or the Hewlett-Packard internal order-tracking software project that cost the company so much that several executives were fired. Not to mention the FAA's communications system that, along with the backup designed to prevent such problems, unexpectedly shut down because scheduled maintenance had been missed, causing delayed flights in Southern California. Rather than comb through Associated Press reports, however, we prefer to turn to our readers for their own brushes with failure. Let the celebration begin!

The Invisible DBA

At one of the big clients I'm consulting for, a story is told about an Oracle DBA who worked simultaneously in two different business units, charging each of them fulltime for several years before his mischievous swindle was discovered.

The con artist had a perfectly organized routine: checking in at his first office, logging in on his first workstation, and going for a round of the usual morning chitchat with his colleagues. Then, carefully leaving a half-empty coffee cup on his desk and a spare waistcoat on his chair, he would leave to put in an appearance at the other building. There, he would repeat the same scenario with a second bunch of gullible colleagues.

The story offered no details about how the man's routine was exposed, but it was very instructive, not only because it revealed that big organizations need some sort of a centralized contract management system for external consultants, but also because it exposed the wealth of magic and mystery that floats around the position of DBA.

Who knows the correct incantations that must be chanted to fine-tune a database engine? Who can cast the right redo-log spell for replicating two bases? Who will decipher the mysterious explain plan? Who is the only one allowed to enter the Holy of Holies and come close to the true asset of a company—its data?

Well, though this particular specimen happened to be a crook, Mr. DBA accomplishes all of the above. But what happened here would not have occurred if the man was viewed as a "simple"—and more easily accountable—software engineer.

In conclusion, I offer a note of hope: Current efforts made in the agile movement to de-dramatize the database will probably help us dispel the fog of superstition that surrounds the subject. And the Mr. DBA in your company will cease being an untouchable enchanter and be considered merely the invaluable expert he really is.

David Dossot, Vice President of Technology/Founder
Agile Partner SA , Thionville, France

The Case of the Missing Millions

While consulting for a well-known brokerage firm on a test automation pilot project, I worked with the project manager to develop a business case for the

investment in software and resources. This calculation typically includes a demonstration of time savings resulting from the automated execution of tests, which is multiplied by the cost of test resources, to obtain a return on investment expressed as money.

In this case, I made the argument that the existing test coverage wasn't sufficient to begin with, and so showing a cost savings from reduced resources didn't make sense. It was more important to increase test coverage and therefore quality; after all, the application was used to trade and manage billions of dollars in mutual funds—sums that dwarfed the miserly testing budget. Although the project manager agreed in principle, he was concerned that there was no way to put a dollar value on improved quality and went forward with a classic time- and cost-savings ROI.

Ironically, a disaster proved my point. One day, the project manager's pager began to buzz—a 911 call—and all hands were summoned to an emergency meeting. As a consultant, I wasn't invited, but later that day he gave me the scoop.

It seems that somewhere along the line, after a software update—no one was exactly sure which one, since they happened constantly—the system stopped charging short-term redemption fees. A quick primer for those who aren't experts in mutual funds: A short-term redemption fee is charged if you hold the mutual fund shares less than a minimum amount of time. Some say the fee is designed to prevent market timing; others say it's aimed to recover administrative costs for frequent trades.

Since these fees were part of the fund revenues, and the fund managers were compensated based on the fund's performance, this directly affected the wallets of some very influential and demanding people. Something had to be done!

The initial estimate, based on past experience, was daunting: About $17 million in fees was missing. In a frantic effort to recover as much as possible, the company assigned 20 people to research prior trades, identify those who qualified for the fee, and seek to recover the money from the customers. Some accounts were closed, so the money was lost, but in other cases they were able to recover it.

All in all, after six months of effort by 20 people, about $11 million was eventually recovered, but in many cases it was at a cost of company credibility and customer loyalty. After all, how comfortable would you feel if the company managing your money couldn't keep track of its own income? And how many of us want to pay fees on transactions that occurred months ago?

The flip side? About $6 million in fees, plus $2 million in resources, was lost—all due to inadequate testing of a single software release. Given that the project manager had predicted a total savings from test automation of $1.6 million over the next 10 releases, it was clear where the real return on investment would have been. A single bad release cost eight times the entire automation savings for 10 releases.

As in the case of most costly, embarrassing software failures, no press release was issued, and outside the department that was directly affected, no one knew. Few companies brag about their mistakes; those who do are forced to by disclosure requirements. For example, in 1999, Hershey Foods had to explain a 15 percent drop in quarterly revenue to its shareholders when a software glitch prevented the shipment of enough candy for the Halloween season.

The good news? Today, the brokerage firm has an automated test library of more than 3,000 trades that are executed for each and every release of its mutual funds system. The bad news? The project manager has left, and testing is being sent offshore to teams that have never heard of the missing millions and may someday decide that they can't afford to keep the automation effort current or that 3,000 trades aren't needed for testing. Thus, as others have found, they may be doomed to repeat history.

Linda Hayes
Chief Technology Officer
Worksoft
Dallas, Texas
 

The Trouble with Backups

A number of years ago, I developed a system to help a multinational corporation track fraud. About a year after I delivered the system, I received a call asking me to look at a problem the corporation was having with the Clipper-based application. After a day of trying to repair the data files that had somehow been corrupted, I gave up and asked the company's tech support people to restore the tables to their most recent usable state, which at that point was several days prior, owing to the time lag in the client calling me.

The next day I came into work, expecting the files to be restored to help the client determine the state of the data and decide where to begin catching up with the recently received fraud reports. When I saw that the files were unchanged, I called tech support to discover the hold-up. They told me they were "working on it." A few hours later, I called again, and this time they 'fessed up: Not a single restorable backup tape could be found.

They told me that backups were run every night, and that they dutifully checked that the task had run before they replaced the tape and relabeled it. But apparently no one had ever attempted a restore—even for testing purposes—to ensure that the backup tapes were useful.

The client was justifiably upset, but instead of taking it out on his tech support team, he turned this horror story into a nightmare for me, threatening to sic his legal department on me if I even wrote him an invoice.

Brian E. Hoffman
New York, N.Y.

Frankenware

It had all the makings of software sophistication—a clever name and components harvested from the best and brightest. PhD (Project History Database) was a grand experiment on the table of software science, but the product that stared back with glassy, flat eyes made us all wonder if we had unwittingly created a monster.

An internal project intended to help IT staff manage our own projects, PhD was the CIO's brainchild. It began on the premise of simplicity, but by the time other eager hands dipped into vials of pickled parts, the project began to take on a life of its own. Automated e-mails pulsed behind each gray button. Configurable options winked slyly on the surface, coiling quietly below in the twisted bowels of infected code. Eleventh-hour interfaces showed the telltale signs of botched surgery—parts grafted to parts like potted programming meat.

We should have raised our torches as an angry mob and run the beast's creator out of town, but we settled on a punishment horrible enough to fit the crime. We, the monster's creators, are now forced to use it on a daily basis.

Donna L. Davis
Greenville, N.C.

Nightmare in Pastel

My worst horror story came from a client who didn't want to let go of the past. This client had a system developed in FileMaker Pro, and the company originally had seven stand-alone locations that didn't communicate with each other at all. When we finally convinced them to go online, they still wanted to stick with FileMaker Pro. So the first part of the horror story was the nightmare of consolidating records and making sure nothing was duplicated, but we had nothing that could identify duplicate information. When we tried to use a Social Security number as a unique record identifier, we found out that the employees had either been making them up or entering "No SSN." This took a lot of data massaging to get a database that made sense.

I thought that was the worst of it, until I started designing improvements. To work on this project, I had to spend four out of five days a week programming at the client's location. The owner made programming decisions based on whims such as "I don't think the employees will remember to do that." So, changes to the program had to be made gradually to give the employees time to adjust. Even report layouts had to be done over time. If I moved a column or inserted a new one, I had to put a comment on the page to spell out the differences to the employees.

One of the final problems I had with this client didn't have to do with layout or programming. They didn't like the color scheme (blues and grays), so I had to redesign the whole program in what came to be known as "Easter egg colors." Report titles, buttons—everything was pastel pink, green, yellow and blue. After a while, it hurt my head to even look at the screen.

This project went on for five agonizing years with no final product—it just kept mutating and never finished. We finally dropped the client, so I don't know how the project finally ended up. But my erstwhile enthusiasm for Easter egg hunts was never the same.

Melanie Hardy, Programmer
Biz-Soft, Oviedo, Fla.

Vehicular Net-Slaughter

The warm Carolina sun shone brightly over the Solid Waste Transfer Station, which had rarely garnered attention from the rubbish-propagating citizenry—until now. After a lengthy and painful software implementation, the county IT department supporting it had finally heaved a sigh of relief. Project complete: maintenance mode. The implementation had been wrought with obstacles, complicated by the facility's remote location. A tower had to be erected to support a wireless connection, and fiber had been run between the buildings.

Then, in a matter of seconds—surely played out again and again in the tormented mind of the poor soul whose momentary distraction left devastating results—the happy operation was reduced to rubble. At 9:30 a.m. on that fateful day, a driver pressed the lever to release his load, causing the enormous truck bed to lift high, tilting and releasing its contents. Maybe the driver's favorite song was playing on the radio, or perhaps his cell phone rang, but the next thing he saw through his rear-view mirror was the reflection of a twisted transformer mount and an enormous tower crashing like a felled tree. He had failed to lower his truck bed before driving off and it snagged the phone and fiber cables above, acting like a weapon of net destruction.

Thankfully, beauty rose from the ashes of mangled cables and splintered wood: That particular design flaw—suspending cables above (rather than burying them below)—was not repeated.

Donna L. Davis
Greenville, N.C.

Crazed Beagles—or Cerberus?

For its small size, the company was loaded with an absurd amount of sheer developer talent. The customer base for our design software was small—our competitor had a hammerlock on the market—but had been fanatically loyal for years. Now was the time to break out of the competitive pack with a drop-dead-gorgeous product that would put us on the map and in the money. We had the skills, we had the tools, we had the vision.

Well, we had several visions. Quite a few, in fact. OK, OK, we had so many we needed eyes like dragonflies'.

I was solidly in Faction One, the Steve Jobs Wannabes. We were afire with the power of 3D graphics, virtual reality and direct-manipulation metaphors. We conceived of a product so revolutionary that we had trouble even conveying the idea to working engineers in our target market. Who cared—it was so cool, they were bound to buy it! In a monopoly market, we figured, the only way to make progress was with an insanely great product. Plus, we were sure to get a standing ovation at SIGGRAPH, which was what really counted.

Faction Two's motto was "Move Over, Let the Users Drive." They didn't want to embark on any project without a demonstrated need and a detailed specification. (Actually, they wanted four specs for every project: requirements, functional, design and implementation. Never did figure out what the distinctions were.) Meticulous developers all, their product features tended to be pedestrian, but bug-free. The trouble was that our company was too tiny to afford much more research than "Hey, call Mark at Isthmus Design and see if he'd like it." And, as always, the users could rarely suggest anything more useful than "Like this. Only better."

Faction Three swung a big clout club: the absentee owner. Blessed with domain knowledge but cursed by a deep and abiding ignorance of how to build software, his refrain during his twice-yearly visits was always "New product, sure. But first, we need to fix these bugs. And maybe put these little features in the old product, hmm? Some upgrade revenue would be nice." Infamously frugal—his favorite tactic was to stiff our landlords for our office rent—he never quite grasped the idea of hiring enough extra staff to fix the old and build the new. (I found out later that's why they hired me—as a maintenance programmer—but by then I was yo-ho-hoing with the Faction One Pirates and pretty much untouchable.)

We definitely could have done without Faction Four, which single-handedly cost us an entire quarter of time-to-market. Newly hired to replace a Faction One guy who'd finally quit in frustration, Mr. Four mined our brains for enough domain knowledge to sound glib, opened a private hailing channel to management, and promised them a new product in 90 days if they'd put the rest of us on maintenance tasks and let him take the helm. Ninety-one days later, he was on the street again, and we were back to square one.

It was like being leashed amid a pack of crazed beagles, all simultaneously attempting to bolt toward every point of the compass. A great deal of ground got covered—or at least pawed—but the lunges in opposite directions averaged out so perfectly that no progress occurred in any direction. In the end, we were left with nothing but a few steaming piles of product proposals, the market window closed forever, and the talent slipped their collars for greener pastures.

Come to think of it, though, nobody has yet come out with Faction One's product. Hmm. You know, those guys all still live here in town …

Rick Wayne, New & Noteworthy Editor
Madison, Wisc.

Read the Appendix

My horror story stars a supposedly off-the-shelf document repository system that was sold to a Fortune 100 customer by being positioned as a human resources information system, since the HR department was buying. It was rife with very strange requirements—from a document repository point of view. The staff struggled to pull it all together, but a few days before beta test, the customer asked, "Oh, now in this release, we get the feature where Criminal Background Checks can be listed as being present but not viewed?" The stunned development project manager replied, "Um, there's no such feature in the requirements."

It turns out that the brilliant analyst who wrote the requirements document (and then promptly left the company) managed to leave this feature out of the functional requirements document—at least the main body of it. But if one looks in the appendix where each document type is described, there, in the comments field for "Criminal History," one also finds verbiage to the effect: "This is the one [field] that can only be listed but cannot be viewed by non-admin users."

Name Withheld for Witness Protection