For Lack of a Bolt
NASA orbited the first Television Infra-Red Observation Satellite (TIROS) in 1960, followed by increasingly complex satellites through the NOAA-KLM series jointly controlled by the National Oceanic and Atmospheric Administration. The current satellites have a minimum two-year lifetime, which requires a fairly steady stream of replacement birds that, it seems, is a difficult pace to maintain.
Two satellites in the same polar orbit observe each point on the earth every 12 hours, so they're built in pairs. The project manager for the new NOAA-N and N' (also N-Prime) satellite pair convened a risk-management meeting by asking attendees to brainstorm possible failure modes and create procedures to prevent those mishaps. He explicitly ruled out things that simply can't happen, such as dropping a satellite on the floor. The 6 September 2003 phone call describing the situation in Figure 1 came as a nasty surprise.
During much of its assembly, a satellite stands vertically on its base atop a handling cart, but some operations require other alignments. The satellite's structure has enough rigidity to support it horizontally from its base, which terminates in the flight adapter that joins the satellite to its booster and serves as a robust mechanical connection during construction and handling.
The white turnover cart (TOC) in Figure 1 includes a hinged plate that supports the satellite, with a bearing that rotates the satellite around its long axis. Actuators pivot the plate between vertical and horizontal positions. Because Lockheed-Martin Space Systems Company (LMSSC) also builds the Defense Meteorological Satellite Program (DMSP) satellites, which are similar to the NOAA N-series birds, in the same building, the two projects share a common set of TOCs. Even though the satellites both ride Titan-II boosters into orbit, they have different flight adapters and thus require different adapters between their bases and the TOC.
You can feel it coming, can't you?
The DMSP Program decided to use this TOC for its activities ... . The reconfiguration was interrupted part way through the process of the TIROS adapter ring removal, in order to install the DMSP adapter ... . This change in plan left the TIROS adapter ring sitting on the TIROS TOC with its 24 attachment bolts removed.
The TOC already had a "red tag" due to a damaged floor jack. The Board discovered that [n]o red tag nor any other indication was added to the TIROS TOC to indicate the incomplete configuration. None of this was communicated to the TIROS folks ... , because the over-riding philosophy was that each user was required to verify or ensure the [Ground Support Equipment] configuration was appropriate for its own specific use each time it was used.
The TIROS crew repaired the jack, hoisted the satellite with an overhead crane, lowered it on the TOC adapter, then installed and torqued 44 bolts between the satellite's flight adapter and the TOC adapter. During the operation, the Technician Supervisor commented that there were empty bolt holes, a conversation that was overheard by several of the technicians. The team and the RTE [Responsible Test Engineer] in particular dismissed the comment and did not pursue the issue further.
A total of 88 bolts secure the satellite's flight adapter to the booster, but because ground handling imposes far less stress than launching, standard practice calls for installing only half the bolts. That left 44 "normally empty" holes in the same area as the 24 missing bolts, obscuring the problem while simplifying the procedure: ...members of the crew recognized that, "This was the smoothest this operation has ever gone."
With N-Prime standing atop the TOC, the crew activated the pivot motors and, when the TOC plate reached an angle of 13 degrees, the satellite simply slid off the plate and punched a neat crescent dent in the TOC on its way to the floor. Nobody got hurt, but it could have been much, much worse, as the Nickel-Cadmium batteries were fully charged, the propulsion system was pressurized, and the separation band was tensioned.
As you might expect, the Board found [t]here were missed opportunities that could have averted this mishap.
