Denver baggage system free essay sample

Analysis of the Denver International Airport baggage system Michael Schloh Dan Stearns, advisor Title Abstract Contents Introduction Reasons For Automation Functionality Of Original BAE Design Problems and Solutions System Complexity Comparative Functionality Opening Delays Financial Hardship Summary Glossary References THE DENVER INTERNATIONAL AIRPORT AUTOMATED BAGGAGE HANDLING SYSTEM by Michael Schloh Computer Science Department School of Engineering California Polytechnic State University 1996 Date Submitted: February 16, 1996 Advisor: Daniel Stearns ABSTRACT This document discusses events at the new Denver International Airport that resulted in opening delays of the airport. The scope is limited to the automated baggage handling system, which was the primary source of failure warranting the airports several opening delays. Analysis of the failing system is comprehensive. Research is conducted using a variety of sources. The final report is published on the worldwide web. CONTENTS Introduction 1 Reasons For Automation 2 Functionality Of Original BAE Design 3 Problems and Solutions 6 System Complexity 12 Comparative Functionality 14 Opening Delays 15 Financial Hardship 17 Summary 19 Glossary 20 References 21 INTRODUCTION This research concerns the automated baggage handling system which was built by BAE Automated Systems, Incorporated of Carrollton, Texas for the Denver International Airport. The analysis of this system provides an important topic of study. From the baggage systems failure, principles of computer systems were clarified and many lessons were learned or relearned by those involved in the BAE project. While there are a variety of issues to learn from the many operations in the construction of the Denver International Airport, focus is placed on the baggage system itself. Some less relevant chapters serve to inform the reader of the occurrences that were influencing the timing and financial properties of the baggage system work as it was built. Reasons For Automation begins by describing how it was decided that Denver International Airport would have an automated baggage handling system. A short review of the history of Denver International Airport in its planning stage illustrates the options that Denver had to choose from. Functionality Of Original BAE Design describes how the baggage system was intended to work. It is a detailed explanation of what makes the system work. Here, parts of computer machinery are itemized, and specifications are explained. Problems and Solutions is the largest chapter and describes what went wrong, and how the problems were solved. This chapter includes descriptions of mistakes made in both the design and construction of the system. Obvious problems such as paint covered optical scanners are explained. Less understandable problems such as the puzzling line balancing problem receive attention. Problems with scheduling and complexity are quickly reviewed, since both topics receive chapters of their own later in the report. System Complexity was likely the predominant cause of the baggage systems failure. Surely many current control and information systems projects in the design phase could be simplified at great benefit to the construction and maintenance of them. The BAE designs failure provides more than enough incentive for other engineers to redesign or simplify a complex design when success of the whole system is at stake. Present industrial trends are horrific. By some estimates, 75 percent of all information systems projects are plagued with quality problems, and only 1 percent of them are completed on time. Comparative Functionality explains how the baggage system really worked when the Denver International Airport finally opened on February 28, 1995. Needless to say, its performance was quite different from what the systems original specification called for. This chapter, in a sense, is a dream versus reality comparison. Opening Delays tells how the project schedule was affected by the profound complexity of the design. The confusion resulted in a prolonged testing phase, reducing the process to solving by trial and error. Systems analysts and engineers hacked together solutions as they went. This unappealing course did the job at the expense of time. Financial Hardship describes the way that the airport was initially funded and the direction of its financing after problems and delays affected its credit. This chapter explains what the city of Denver and airlines did to account for budget deficits and cost overruns. Summary concludes the study with a review of the lessons learned, and how they can be constructive in avoiding similar failures or even worse, larger failures of catastrophic magnitude. REASONS FOR AUTOMATION United Airlines Request Early in the planning stage, United Airlines insisted on an automated high speed baggage system, like the one it operates in San Francisco. After some consideration, Denver agreed that not only would United have an automated high speed baggage handling system, but so would the rest of the airports three concourses. Denver officials had sound reasoning in choosing to install an automated baggage handling system. Before deciding on buying an airport-wide system, Denver officials had previously assumed that each airline would design their own system, according to its own needs. When the airlines failed to produce their own designs, Denver investigated the option of buying a system to service all airlines in a unified manner. When the planners considered a traditional manual baggage handling system using tugs and carts, it appeared to be inadequate for a few reasons. Moving baggage by the traditional system is a labor intensive and expensive process. The tugs are diesel powered and would not have been able to travel through the poorly ventilated underground tunnels due to the high volume of diesel exhaust that would have choked the tug drivers and other workers. Even if ventilation had been installed, the heavy volume of large tugs and carts would have jammed the small tunnels as they passed each other or turned corners. Long Distances An additional concern involved spanning the great distances of the airport. At the Denver International Airport, distance and speed of delivery have especially significant importance because the distances between passengers, planes, gates, ticket counters, concourses, and the terminal are much larger than at other airports. The closest concourse, concourse A, is 1,300 feet away from the passenger terminal. The farthest, concourse C, is a full mile from the terminal. Concourse B itself is . 7 miles long. To keep flights on schedule, speed becomes critical in moving baggage. Furthermore, across such great distances the only direct route for baggage moving is through the underground tunnels, which are incapable of accommodating gas-powered tugs. Taking baggage on tug and cart by route of the runway aprons could take as long as fifty minutes, thereby missing most flights. Glenn Rifkin states, For an airport this size, a conventional baggage system simply wouldnt work. Increased Profits For Airlines The airlines were as disappointed as the city in a traditional manual system. In general, airlines maximize their profits by keeping their planes airborne, not grounded and waiting for baggage. United knows this too well after enduring some of the worst gridlock and bottlenecking in the nation at Denvers Stapleton International Airport. Stapleton frequently ranked fiftieth out of fifty airports rated for on time performance according to Briggs Gamblin, a spokesman for Mayor Webb. United accordingly sought to keep their airplanes in flight and on time by insisting on an automated system in the construction of the new airport. Denver began researching the possibility of an airport-wide automated system, and with BAEs help, planned such a system and sent it to bid. FUNCTIONALITY OF ORIGINAL BAE DESIGN Savior Of Modern Flying When the automated baggage system design for the Denver International Airport was introduced, it was hailed as the savior of modern airport design. Designed by BAE Automated Systems of Carrollton, Texas (previously Boeing Airport Equipment), it allows airport planners to design airports of larger size, using narrow corridors and tunnels for baggage where no tug and cart system can run. Furthermore, it requires none of the manual labor personnel, and can be used as easily in pinpointing the location of baggage as in moving it. The design truly fits its description as the worlds most advanced baggage handling system. It is intended to run faster and more reliable than traditional technology. Its automation is so thorough, that in most cases, baggage offloaded from an aircraft doesnt see a human until it meets with its owner at the baggage claim. The systems speed outperforms even the airports high speed trains. Flyers never have to hover around the baggage terminal waiting for their baggage as with traditional systems, because their baggage arrives at the claim before they do. On departure, their baggage arrives at the aircraft before they do. Other Automated Baggage Systems While the automated baggage system design of Denver International Airport is unique in complexity, technology, and capacity, it is not the worlds first such system. The three other airports that have such systems are San Francisco International Airport, Rhein-Main International Airport in Frankfurt, and Franz Joseph Strauss Airport in Munich. The major distinctions that separate Denvers design are size and complexity. While Denvers design is integrated to sort baggage from all airlines throughout the whole airport and deliver over a thousand bags per minute, the other airports use systems that are localized to much smaller baggage loops and offer less capacity. San Franciscos system is ten times smaller and handles fourteen times less in speed and capacity. The system in Frankfurt runs on trays and conveyor belts rather than Denvers high speed telecars and is three times smaller in size. Munichs automated design is similar to Denvers but far less complex. High Speed Denvers baggage system design calls for replacing the traditional slow conveyor belts with telecars that roll freely on underground tracks at more than three times the speed. A telecar that is loading baggage rolls at 4. 5 miles per hour. A telecar that is unloading its baggage rolls at 8. 5 miles per hour. A telecar in transit rolls at a fast 19 miles per hour. Each track can handle 60 telecars per minute. It is the combination of using Denver International Airports underground tunnel network and swift speeds that allows all baggage to move between any concourse and the airport terminal in less than nine minutes. In Uniteds concourse B, transfer baggage moves between any two gates in under six minutes. According to Briggs Gamblin, a spokesman for Denver Mayor Wellington Webb, the systems high speed nature is intended to shave minutes off the turnaround time of each arriving or departing flight. Components The BAE design includes a number of high-tech components. It calls for 300 486-class computers distributed in eight control rooms, a Raima Corp. database running on a Netframe Systems fault-tolerant NF250 server, a high-speed fiber-optic ethernet network, 14 million feet of wiring, 56 laser arrays, 400 frequency readers, 22 miles of track, 6 miles of conveyor belts, 3,100 standard telecars, 450 oversized telecars, 10,000 motors, and 92 PLCs to control motors and track switches. With so much equipment serving such a large area, the Denver International Airports baggage system is the worlds largest. This project is of the same magnitude as the Panama Canal or the English Channel Tunnel, said Mayor Webb. The systems total cost is $193 million dollars. Baggage Handling Process Because of the revolutionary automated baggage system, the process of handling baggage is unique at Denver International Airport. At check-in, agents stick glue-backed bar code labels on baggage, identifying the bags owner, flight number, final destination, and intermediate connections and airlines. Instead of printed bar code tags, Uniteds portion of the system uses photocells that serve the same purpose. The check-in agent then puts the bag on a conveyor belt. Since no baggage can move without a telecar holding it, a system exists for dealing with telecar allocation. Empty car management software is the heart of the allocation system, dispatching empty telecars to where the tracking computers anticipate they will be needed. The computers sense changes in demand by measuring the flow of passengers throughout the airport. During peak times, all 3,550 telecars are available for moving baggage. When an empty telecar arrives, the conveyor belt holding the bag advances. Then a type of high-speed luggage bowling machine flings the bag at a T-intersection just as the telecar moves by, catching the bag in its fiberglass tray. Each telecar has a tray for this purpose that tilts into three positions for automatically loading, carrying, and unloading its baggage. In Denver International Airports system, telecars do not stop for loading or unloading, they only slow. This type of Dynamic loading increases handling capacity and saves energy as well. Before the telecar speeds away, a laser scanner similar to those used in grocery stores reads the bar code tag on the bags handle and associates the bag with its telecar. These laser scanners are triggered by photo-electric sensors that detect a telecars presence. Telecars pass photo-electric sensors every 150 to 200 feet of track. The computer that scans the bar code tags then sends information to a BAE sortation computer that translates it by using a look up table to match the flight number with the appropriate gate. A tracking computer guides the telecar to its destination by communicating with the hockey puck-sized radio transponders mounted on the side of each telecar. The telecars are able to move on the tracks by linear induction motors, or LIMs, which are mounted periodically on the tracks, and push the telecars along. A metal fin on the bottom of each telecar slides through each induction motor gaining impulse as it goes. Telecars merge with other telecar traffic and exit to unload stations by computers which control PLCs, or programmable logic controllers. The computer tracking a specific telecar directs it by communicating with PLCs that are responsible for causing track switches. Tracking Baggage As the telecars roll, the tracking computers monitor each of the systems thousands of radio transponderswhich emit millions of messages per second. The computers must also track all gate assignments so that the telecars can be re-routed if a change is made. The tracking computers can also re-route bags to special inspection stations, including one that is bomb proof. The same computers must keep track of obstructions or failures as well, so that telecars can automatically detour around a stalled vehicle or jammed track. Oversized Baggage In addition to standard-sized baggage, the system can also accommodate nonstandard-sized baggage on oversized telecars that measure 6. 5 feet long by 4 feet wide. The oversized telecars are essentially double-length standard telecars. They are meant for non-standard size baggage which in Denver typically tends to be skis and golf bags. The oversized telecars navigate through twists, turns, and switches the same way the standard telecars do. Security Impressingly, the system can work in full capacity for 18 hours every day at a 99. 5 percent efficiency rate. Two counter-circulating closed-loop tracks with multiple routing connections provide for future expansion and add redundancy to guard against unanticipated problems. To protect against malice that could theoretically shut down the whole airport by halting the flow of baggage, tight computer security is built into the baggage system. The system has strict access privileges for workers, and its command center is well guarded and locked behind steel doors. Despite BAEs conflicting advice, the entire automated baggage system is run by DIAs information systems staff of 18 employees, according to Ivan Drinks, director of MIS for both Stapleton and Denver International Airport. Object-Oriented Architecture Fortunately, the automated baggage handling system illustrates the principle of object oriented design beautifully. It sends messages to objects (the telecars), which respond by returning other objects (baggage and empty telecars) to the sender. Its real-time software was programmed in OS/2 and intended to run on OS/2 version 2. 0. Decentralized computing allows the baggage system to operate independently of the airports information systems department. The only dependence within the system involves coordination with the airlines flight reservation and information systems. PROBLEMS AND SOLUTIONS Denvers Baggage Problems The Denver International Airports automated baggage system experienced such horrific problems that most with an opinion on the matter are thrilled to elaborate on their sense of what went wrong. It seemed that what could go wrong, did go wrong. Even the signs directing passengers to the baggage claim led to a concrete wall. Unfortunately, analyzing the true nature of the systems faults is not an easy task. Problems were so widespread, that possibly no small number of reasons can alone account for the chaotic performance in the systems early testing. Insight can be found in examining the accounts of some key people who were involved in the baggage project. Expert Opinions In response to criticism after the third opening delay, BAE president Gene DiFonso explained, We simply ran out of test time because of changes requested by the airlines, problems working around other vendors, and failures in the airports electrical power supply. Denver aviation director James C. DeLong maintained that baggage software glitches and electrical supply harmonics were late and unexpected obstacles to opening the Denver International Airport. According to David Hughes of Aviation Week Space Technology, contributing factors to the baggage systems problems included concrete mechanical, electrical, and software flaws. William B. Scott of Aviation Week Space Technology believed that the systems troubles originated in more fundamental miscalculations such as overall system complexity, underestimation of tasks, a steady stream of changes requested by both airline and Denver officials, and politics. Politics Political issues were a surprising obstacle in the progress of the automated baggage system design and installation. George Rolf, an urban planning professor from the University of Washington, said that publicly run projects like Denver International Airport encounter problems because you have two distinct processes going on, one political and the other technical, and they have little to do with one another. One example of this claim is Denvers refusal to award the job of operating the baggage system to BAE, the only company that well understood it. The basis of this decision revolved around political but impractical ideals. Essentially, Denver officials suspected that BAE would not hire enough minorities and women, although BAE said they would. Richard Woodbury wrote, In the wake of political infighting over who should get the lucrative contract, it went to an outsider, Aircraft Service International of Miami, which has had to race to fathom the system in a few months. A Denver insider declared, It was raw greed. Everyone wanted a piece of the contract moneys. The city lost control at the outset, and the project was destined to run amuck. Further political problems ran through the entire Denver International Airport construction in the presence of rhetoric and false assurances to the bond market. Some of the statements made by Denver in defense of construction delays and practices bordered the lines of legality. Mike Boyd, an analyst who heads Aviation Systems Research Corporation in Golden, Colorado said, This is an airport built for politicians, not for airlines. When you look at the numbers and what theyre telling bond houses, it is absolutely shocking. None of the significant numbers that the city has been putting out since the airport was started have held true. Other political troubles included Denvers alleged falsifying of temporary certificates of occupancy (TCOs) in the midst of the baggage system crisis to appease the airlines, and a lawsuit with the Park Hill Neighborhood Association barring a partial airport opening. Consequently, in January of 1994, both the Justice Department and the Securities and Exchange Commission subpoenaed key Denver International Airport documents. In February of 1994, the U. S. attorneys office sent investigators to Denver to interview city officials and probe into alleged wrongdoings. In August of 1994, a federal grand jury began investigating the Denver International Airport for fraudulent contracting, trading, testing, and construction financing practices. In late October of 1994, a congressional auditing agency became involved in Denver International Airports financial woes. The General Accounting Office (GAO) reported that despite Denvers delays and losses, the citys chances of avoiding default were good. Technologically Advanced The BAE design is technologically advanced. According to Richard de Neufville, it is not the next generation of baggage system, it is more like a jump from third to fifth or sixth generation. Unfortunately, BAE misused its technological advantage by expecting spectacular performance from the system components, and not allowing them a proper margin of error. The components were expected to perform to their highest theoretical capabilities. Bruce Van Zandt, operations manager for the backbone communications network at Denver International Airport stated, The system pushed the envelope of technology. The components that were put into the system were run right to the limit of what they were designed for. When any of the components failed in this respect, others failed as well due to the systems inherently tight coupling. Planning BAE, DiFonso said, was originally contracted by United in the fall of 1991 to build a baggage system specifically for United Airlines at the new Denver International Airport. The airline, he said, was concerned that after several years into the project, the city still had not contracted for a baggage system. Indeed, Denvers baggage system design was an afterthought to the construction of the airport. The BAE system was detailed well after construction of Denver International Airport had begun. When construction of the automated baggage system finally began, problems arose due to the constraints of the buildings and structures which would contain the baggage systems tracks and other components. Unfortunately, the system had to fit into the underground tunnels and available space given the challenging and unrelated Denver International Airport construction plans. Tight geometry resulted in additional construction difficulties. Telecars had to make unreasonably sharp turns on tracks shoehorned into corners at considerable inconvenience. According to Bernie Knill, an obvious solution to such poor planning techniques entails designing the baggage handling system with the building, and installing the system as the surrounding structure is being built. Schedule BAE officials said that a timetable for the opening of the airport was never realistic and should have taken potential problems into account. When asked about the ambitious timeline, one BAE official responded, We knew that was not long enough and we said so. Its a job that ought to take twice as long. While the media hammered BAE for their role in the delays, BAE vice president of engineering Ralph Doughty voiced his frustration. Its a 3-4 year job we were asked to do in 2 years, he said. Denver Aviation Director James C. DeLong offered the explanation, We had a project that should have taken seven years and we tried to do it in four years. We just misjudged. Well probably do it in five. As the project fell more and more behind, human error became a factor due to a more truncated training and testing period. Requirements Modifications and Other Changes When BAE accepted the job, no changes to the project were anticipated, DiFonso said. However, once BAEs work had begun, Denver officials often altered plans and timetables without consulting either the airlines or BAE. Even worse, when changes were made to one part of the system, it was not clearly understood how the changes would affect the system as a whole. To reduce its construction costs, United decided to remove an entire loop from its own ambitious design for concourse B. Rather than two complete loops of track, United wanted just one. This change shaved $20 million off the systems price, but required a complicated and untimely redesign. Other changes were made such as relocation of outside stations, addition of a mezzanine baggage platform, and Continentals request for a larger baggage link. As the project matured, it grew in size and complexity. Design changes increased the systems technical difficulties that consistently hampered progress. When BAE learned that the centralized systems faults ran through the rest of its tightly coupled subsystems, they chose to decentralize all of the tracking and sorting computers. Such major design changes deserved review of alternate courses. However, due to the condensed development and testing schedule, on the fly design changes that typically require major design alterations were treated with minor patchwork Chaos The first time that BAE ran the baggage system for performance testing, the resulting chaos was sobering. In March of 1994, the installation staff ran the BAE system for several media groups. Faults throughout the entire baggage system destroyed bags and flung suitcases out of telecars. The next day, phrases like bags were literally chewed up, and clothing and other personal belongings flying through the air hit newspapers. Telecars jumped tracks and crashed into each other. Suitcases went flying like popcorn kernels, some of them breaking in half, spewing underwear in every direction. When the telecars crashed into one another they bent rails and disgorged clothing from suitcases. Others jammed or mysteriously failed to appear when summoned. Telecars crashed into each other especially frequently at intersections. Many dumped their baggage off at the wrong place. Some telecars became jammed by the very clothing they were carrying. As the telecars flung their bags off or ripped them open, the clothing clogged the telecar rails, halting traffic and crashing other telecars in back. Most telecars holding bags with unreadable bar codes were routed to holding stations. Other telecars that knew were they were going collided with telecars that couldnt remember. On May 2, 1994, DiFonso addressed the situation, and stated that the system was not malfunctioning, it just hadnt been fully tested yet. BAE officials blamed the mutilation and other problems not on a defective design, but on software glitches, and mechanical failures. They found one reason for baggage mutilation involved the airport personnel. When workers placed bags on the conveyor belts upright, the system frequently jammed or shredded the bags. When the bags were placed correctly, laying flat, the performance improved. BAE found many design culprits and appropriately made changes. Slowly, BAE improved the systems general performance. Unfortunately, in August of 1994, the systems performance was still poor. Even during planning of the alternative tug and cart baggage system, telecars continued to collide and fall off their tracks. In late August, Glen Rifkin of Forbes wrote, Throughout the day, workers are seen unclogging tracks lined with bags that have been cut in half. Morale was low among the installation crew. When asked how the test bags were damaged, one worker replied in mock horror, Its not eatin bags. A truck ran over these outside. Software Ginger Evans, director of engineering for Denver International Airport, claimed that BAE didnt pay enough attention to the programming issues early enough in the design process. She believed that alleged troubles with building access or mechanical issues werent the problem. Its that the programming is not done, she said. She faults BAE for this inadequacy. Others contend that many problems of mechanical nature originated in the buggy software. According to Glenn Rifkin of Forbes, software sent out carts too early or too late. Robert L. Scheier of PC Week alleged that it was the systems software problems that resulted in the airports 3,550 baggage telecars crashing into each other or becoming stranded along its 22 miles of track. BAE president Gene DiFonso contested allegations of faulty software playing the central role in the systems horrific performance by stating that Software was not the major problem. It was an electromechanical problem. The system was stutter-stepping because the electromechanical side wasnt fully up to the softwares capability. However, DiFonso admitted that program code had been a nightmare at times. He revealed that the burden of writing code for establishing and maintaining communication with the airlines reservation systems was heavy. Particularly challenging was the duty of connecting with Uniteds Apollo reservation computers. A definite element in the disarray of the communication software was the process of language translation, since BAEs computers had to converse in the same software language as of each airline. Such translation work is painstaking and often laden with bugs. While writing code for the communication, tracking, and other numerous applications, the software grew more complicated. As a consequence, the code completion agenda experienced the threat of becoming unmanageable due to escalating levels of complexity. By principle, as program code grows in complexity, it becomes increasingly hard to track or understand (see Complexity Of the System). Instances of systems code delaying the opening of large projects abound. For example, the English Channel Tunnel was delayed for about a year by problems with more than three million lines of code. Only adding to confusion, applications of such size typically borrow from a number of object code libraries and other resources. As Bjarne Strousoup noted in 1987, No major program is ever written in the programming language as described in its basic language manual. Libraries of all sorts are used and often determine the structure of the program. Finding the origin of a glitch can consequently be nearly impossible. A giant project held hostage by troublesome software code and insufficient testing is the technologists worst nightmare. When troubles arose with the Denver baggage systems complicated code, BAE programmers had to customize the software to handle each individual software related problem. This process rudely resulted in code hacking. If the baggage handling system has all of its problems solved, it will be via hack-o-rama, wrote Larry OBrian. System Testing According to John Dodge, 75 percent of all information systems projects are plagued by quality problems, and only 1 percent of the projects are completed on time. Dodge cites insufficient software testing as the most frequent culprit and describes it as one of the thorniest client/server issues. Munich officials had advised Denver to leave plenty of time and resources for testing. At the Munich airport, where a smaller automated baggage system sorts baggage, engineers spent two years testing the system. In addition, the system was up and running 24 hours a day for six months before the airport even opened. The Munich officials said that the Denver staff did not heed their advice. Although BAE had tried to leave sufficient time for testing, they were constrained by their promises of a quick pace in developing the system. Moreover, troubleshooting the maze of software was a slow process. According to DiFonso himself, Underestimating the time required to discover problems, fix them, and retest, was the main reason for the opening delays. Testing the systems mechanical side was