Hacker News new | past | comments | ask | show | jobs | submit login
All American Airlines Flights Grounded, Experiencing Nationwide Computer Outage (techcrunch.com)
155 points by jstreebin on Apr 16, 2013 | hide | past | web | favorite | 135 comments

I'm at DFW right now heading to LAX on AA. Computers just came up and they are going to begin boarding shortly. Lots of unhappy travelers here as you can imagine I'm sure.


To follow up. It's been about an hour since the computers came back up in the terminal and I'm finally in my seat on the plane. About 3 hours after it was supposed to depart.

There were several canceled flights in my terminal and no other flights going out so people had to stay overnight. Most were very mad as American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess.

I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what".

> I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what".

Yeah, they'll never fly American again no matter what...until it has the lowest price ticket for the time/destination they want.

Another update. I'm still here. We are parked at the gate waiting for additional fuel before we can leave. Not sure why they boarded us just to have us sit here for another hour (or more - no clue how long this will end up taking). The good news is the plane has AC power in my row so I was able to build the navigation for a site I'm working on.

I think they board the plane as quickly as possible so they can get in the queue to taxi to a runway. Until the door is closed, you do not get a place in line to take off.

>Not sure why they boarded us just to have us sit here for another hour (or more - no clue how long this will end up taking).

Every airline seems to have been doing this for decades now, and it boggles my mind why they do this. Do they think that getting people "on board" is demonstrating "progress! breakthroughs! we're moving forward folks!" or something and it's supposed to make us suddenly happy?

I'd so much rather spend the extra 2 hours at the gate than crammed into a 737 for the same amount of time, especially when they decide that they can't turn on the AC (when I was little, we were stuck in a fully packed 747 for 4 hours with no AC. They opened the front and back doors b/c it was so unbearable in there...)

The flight is considered "departed" from a time-tracking perspective when the boarding doors are closed. If your flight is scheduled at 2pm and they close the doors at 2pm, even if it takes you until 8pm to get in the air, your flight "left on time".

This is a perfect example of "be careful what you measure". Customers don't care when they leave, they care when they arrive, but, since the FAA measures[1] on-time departures, that is what has been optimized for, to the detriment of customers.

1. I would be willing to bet the FAA measures this at the request of the airlines. Measuring arrival time has many variables they can't control, whereas departure, as defined, is mostly controlled by the airlines.

It's because that's one of the metrics which is tracked and, therefore, optimized for, even at the detriment to the core business.

Departure slots. When one comes up you have to be ready to go immediately.

This is why.

Having spent time in the military, it makes sense to me. Hurry up and wait :) I actually like that way of doing things.

Boarded a flight recently and fell asleep at once after sitting down. Woke up three hours later, plane was descending. It was supposed to be a 1.5h flight. Oh well.

In the EU the airlines are legally obligated to provide drink and food if the flight is delayed over a certain amount of time. I believe once you're on the plane/tarmac that clock stops ticking.

> American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess.

Why are they not subject to a class action lawsuit? The problems rather obviously seem to be American's fault.

Seems like some enterprising lawyer should buy a ticket, drive to the airport, go to the AA gates, and start handing out business cards to everyone in sight.

> Why are they not subject to a class action lawsuit?

Uh, because it takes time to file those.

Also, I suspect, because they have binding arbitration and anti-class-action clauses in the fine print of ticket purchase agreements.

Such clauses are of dubious legality.

Why are they not subject to a class action lawsuit? The problems rather obviously seem to be American's fault.

Read the fine print on your ticket. You're not promised perfection.

There's no regulation per se, it's whatever is in the fineprint when you purchase your ticket. They typically wave any responsibility if the reason for the delay is any of the following :

bad weather, acts of God, terrorist activities, war, strikes, "any shortage of labor, fuel or facilities” and “any event not reasonably foreseen, anticipated or predicted”

This could fall under "not reasonably foreseen" or maybe act of God for the very religious :).

> go to the AA gates

You have to be a ticketed passenger to get _to_ the gates.

Schedule a meeting in the AA Admiral's Club. The agents will give you a pass to get through security without a ticket.

Then buy a ticket. It'd be a worthwhile expenditure.

Great answer (but if the computers are down, can he buy a ticket?).

Doesn't have to be a ticket on AA.

And it can be fully refundable.

It'd be worth the cost of a cheap ticket.

Problem is: the reservation system's down.

For sake of argument, would you sue AA if just one flight was delayed 3 hours?

The whole point of a class-action lawsuit is to aggregate a large number of people with relatively minor harm into a single lawsuit to avoid heavy per-person overhead. There have been plenty of class-action lawsuits where the harm per person was less than the cost of an airport hotel room for one night.

Because it's a mechanical failure and not an "act of God" they are responsible for any lodging/food for their passengers due to this delay... I don't know if that's a legal thing, but I do know that's how every airline I've ever flown on works.

"Mechanical failure" typically refers to things that are... mechanical in nature, e.g. not software. They are most likely interpreting this under the unforeseen events clause of the ticketing agreement (not sure of the exact wording) to see if they can get away with not paying anybody. If enough of the passengers raise a fuss they will likely offer everyone some sort of voucher, but 90% of people unfortunately will forget about the whole thing 30 minutes after they land.

Yeah that's a pity, it sucks when only the loud obnoxious people get their way...

I fly the DFW to LAX route once a week and have done it on many different carriers (United, AA, and Virgin America). Even though VA offers fewer flights on this route, that has become my sole carrier (even outside of today's events). They handily beat AA on experience, and sell at the exact same ticket price (main cabin, not FC).

Virgin has won me over completely.

Okay, last update. I made it to LAX, picked up my rental car then received an email from American notifying me that my flight was on time - the same flight I just got off of that landed over 4 hours after it should have. I had to laugh at that one.

I like how it says "apologize for ANY inconvenience" as if theres the possibility that all planes being grounded making hundreds of people late for meetings offers any room for there not to be an inconvenience.

Or worse, parents with young children stuck at airports losing their mind.

Or worse than that, missing a meeting and being stuck with other people's young children at airports.

Unsure if you have kids but trust me. It's worse for the parents.

Thats a great post, but the thing is the phrase "we apologize for any inconvenience this may've caused" has a very useful psychological purpose: Diffusion.

Lets assume that I know you are mad, possibly pissed. Possibly so pissed that you are going to move your business away to someone else. By saying "sorry for the inconvenience", I diffuse your feelings....make it sound like "I'm sure you preferred to get there earlier....but no biggie, right?"

By giving you a sincere, heartfelt apology I validate all those reasons of why you are pissed. Gives you justification for your further actions.

Regardless, I don't see why it even matters to AA in this case. Thanks to FAA/Local Airport they have de facto monopolies for certain routes, in reality they don't need to apologize at all. (god, I wish competition existed in air travel).

I don't have anything scheduled until the morning luckily so no big deal for me. I was more inconvenienced by the fact that they don't have enough power outlets to charge devices here (DFW).

To me the tone of that tweet sounded very insincere.

"American said the issue was caused by an inability to get access to its reservations system, called Sabre. The electronic system, often described as the brains of an airline, is responsible for bookings and reservations but also manages a wide variety of functions related to flights, including printing boarding passes, online check-ins, ticketing, and tracking checked bags. [...]

Sabre, meanwhile, said American’s system outage did not come from its own computer systems. Other airlines, including Southwest Airlines and JetBlue, use the reservation system and have not experienced any outages, said Nancy St. Pierre, a spokeswoman for Sabre."



Not really related to this outage, but I think it's interesting to note that Sabre dates back sixty years! The ball got rolling in 1953, with a chance meeting between AA's president and an IBM salesman. IBM was working on a massive air defense computer system at the time, and they decided that a similar system would be good for airline reservations. The system went live in 1960, and has been running continuously ever since.

Indeed, this is one of the more interesting systems evolutions of our time. Its inspiration was the Semi-Automatic Ground Environment (SAGE), a cold war-era air threat assessment computer with real distributed qualities and light gun interfaces! By way of IBM, SAGE begat the Semi-automated Business Research Environment (SABRE, as it was then). It was the first GDS, and has operated in one form or another for 60+ years. Very impressive stuff, and humbling from the perspective of a mere software engineer. Working with such systems, with services interacting sometimes directly with green screens, is a far cry from Facebook and Twitter APIs, or ITA for that matter. This stuff is solid and vital, and the fear (not to mention cost) to manipulate it is astounding.

SABRE was fine. Some of the systems at AA couldn't talk to it. Having seen their systems none of this surprises me.

Somebody probably let an API key expire...

From the NYT article, sounds like it was a connection problem. Which makes you wonder, was it a DoS attack (either end of the connection?) preventing AA from connecting to Sabre?

A similarly described SABRE outage in 1989 seems to have caused minimum trouble, because they had the real capability to work off-line with paper:


> Apparently, no information about reservations and other travel plans was lost, and American said there were no serious disruptions of its 2,300 daily flights.

I can't wrap my head around how extraordinarily complex airline reservations/logistics systems are. I'm kind of surprised you don't see this more often.

Looks like they use Sabre:


from http://www.dallasnews.com/business/airline-industry/20130416...

Looking at Sabre's own timeline, I cringe at what I suspect the codebase must look like. From what I read between the lines here, it looks like neither a carefully maintained legacy system, nor a proper rewrite at any point along the way, but more like 50 years of things bolted onto each other: http://www.sabre.com/home/about/sabre_history

I have the utmost respect for systems that have been running for decades without major rewrites, but having said that, I imagine there is a rock-solid core that has, literally, withstood the test of time, surrounded by semi-compatible systems that accreted around it since the mid-80's, with most current systems only touching this peripheral mass rather than the core services.

I've had contact with many systems that have their cores running on IBM mainframes with peripheral systems running on .NET. I fear those.

Transaction processing on IBM mainframes is typically very robust. It's what they were built to do.

I've been told that various companies have tried to rewrite Sabre on a number of occasions and always failed.

There are other systems that work fine. Sabres real competive advantage is the existing customer base.

United's failed in November, although for a much shorter duration (hour or so). As luck would have it, I was flying United that day, and I'm flying American today =/

It's not that hard of a problem to solve really. Weve come a long way as an industry since SABRE came out 53 years ago. AA's biggest problem is that their IT dept is 80% low grade contractors and they've been unsuccessful in delivering their own solution for what SABRE does. They've been pushing back the sunset of SABRE for over a decade. Many days I'm surprised anything written at AA runs at all.

> They've been pushing back the sunset of SABRE for over a decade.

If AA has relied on SABRE for so long, why would they want to replace it? Cost?

I am surprised no one has mentioned Rule 240. I have, more than once, gone up to the agent when a flight was delayed and asked about this rule. They always hesitate at first, but if you insist, they quickly make provisions to put you on another flight.

This only works when it is the airlines fault and not an issue with weather.

Rule 240: http://en.wikipedia.org/wiki/Rule_240

Putting my tinfoil hat on for a moment...

I wonder if there was a threat against the airline? Perhaps something enough to be credible but not so specific as to be pinpointable, and rather than cause widespread panic by saying "ground every AA flight until we sort it out" (which would freak people out beyond belief), someone figured out that pulling the plug on their mainframe for a few hours would have the same result without the panic. Just some pissed off passengers.

Or maybe I've just been watching too much Homeland. Sigh.

That's just your paranoia. I wish I could speak more freely than I already have (see my other comments) but the fact is, crappy software, written by crappy contractors all tied to a 53 year old system of record.

  Old software + "wish I could say more" = conspiracy

  /defensive mechanism

Comedy of errors... sometimes a very dark comedy...

I can't find the article, but this isn't the first time that an airline has had a problem that grounded all of their flights. There was one in recent memory (last 20 years?) that had a bug where if there were more than 32k changes to the flights in a day it would cause a horrific crash in their system. They never hit it for so long, and then one day, boom all flights just disappeared.

Never ascribe to malice that which is adequately explained by incompetence.

Hanlon's razor.

Agreed... FWIW, I remember something similar happening to both US Airways and United Airlines within a few weeks of each other about 1.5 years ago.

Just a general comment on airlines:

With US-based carriers, expect low prices, inflating awards points/miles, and bottom-level service.

With non-US based carriers, expect higher prices, no awards programs, and excellent service.

This is my experience flying in N. America, Europe, and Asia.

You must not be flying the low-cost airlines in Europe. I have never seen any US-based airline that rivals the combination of low-price and bottom-level service provided by Ryanair. Unfortunately, the low-end airlines are so popular within Europe that it's increasingly hard to avoid them, even if you want to pay more, because they have many of the direct flights (especially to vacation destinations).

Unless your destination is as low-budget as the airlines, it's pretty easy to avoid them. They're mostly the equivalent of a cheap holiday bus.

Also, although I will avoid the EasyJets and Ryanairs as much as possible, for a direct flight the duration of a bus ride it doesn't really make that much difference.

I don't think this is true at all. Europe's low cost airline business is major. Easyjet, Ryanair, Flybe and Jet2 just to name a few. Obviously the service on these carriers is awful but flying between European cities for $50 a flight isn't uncommon. Personally my most consistent good service experiences have been with Air Canada. BA was good too as was Aer Lingus. I've only had two flights with them but I wasn't fond of AA.

Singapore Air is amazing, has a great rewards program, and is affordable.

  | Personally my most consistent good service
  | experiences have been with Air Canada.
I recently saw a headline that Air Canada is the third worst international airline for arriving on time. That said, I had no complaints about my only Air Canada trip (3 connecting flights).

As a Canadian I've used AC and other regional carriers quite a bit. Have to give a shout-out to Porter, if you're flying to the few cities in North America they service, they're top notch. Especially flying into Toronto, they exclusively use the Toronto Island Airport (YTZ), which is much more pleasant and better situated than Pearson (YYZ).

I think AC has gotten better, at least on domestic flights, since they've had more competition.

This was my only issue with Air Canada: my first and only (to date) flight with them was several hours late. It wasn't a problem for me so I think of them as being one of my preferred airlines because of how good everything else was. If the lateness had been a problem I might have seen it differently, though I didn't think the delay was their fault.

I've flown from Ireland to the US and Canada with them several times now and the service has been excellent. Now that you mention it though most of the flights arrived slightly late (no more than 30 mins). As they were trans-atlantic flights and I wasn't in a hurry it didn't bother me.

Even the US-based "legacy" carriers find themselves directly competing with Southwest, which has forced them to gravitate towards the low-cost model. The only thing that differentiates them in such an environment is their award systems. Except Southwest which (ironically?) has the best service and perks now.

It's admittedly been a few years since I flew a US low cost airline. But at least then they where pure luxury compared to Ryanair or Easyjet in Europe.

Huh... Explain how Southwest has done so well then? They are cheap, profitable, and have the best service.

I compared US-based carriers with non-US based carriers. Southwest, a US-based carrier, competes with other US-based carriers. You're right, Southwest is a great choice if you're flying in the US, but that's not what my post is about. You can't compare Southwest with Cathay Pacific or EVA Airlines because they operate totally different routes.

Same thing regarding EasyJet comments, which is another "niche" airline. Try flying EasyJet from LAX to NRT, for example.

Alaska is pretty decent also, but I usually fly in the west coast market. Southwest prices are often not that great, actually, but their service is always good.

Southwest markets as being cheap. Go compare their prices with the legacy carriers and you will often find they are priced about the same.

Thats not always the case.

I've been badly screwed over by Leftouthansa many times, their response: "Nope, we don't care. We look forward to seeing you onboard." [After 2 support tickets, one to complain and then the other to report the unanswered first ticket after 2 months]

I have been treated well by KLM.

They're German. They have a procedure for every imaginable circumstance. And as long as they're following the procedure, there is no problem.

> With non-US based carriers, expect higher prices, no awards programs, and excellent service.

Cough...China Eastern Airlines...cough. And oh my god, Air China. You guys have nothing on inconveniences when compared to the good ole PRC.

I feel for the poor sods thrashing away trying to make this right. Do developers in mission critical areas earn a premium? This could knock a couple of years off your life in stress.

LOL, yah right. No they don't earn more. AA does not pay well. Also most of the legacy systems are maintained by contractors.

What do they earn?

I'm told, "well below market". I've talked to probably 40 programmers there. Only one of them knew what any of the SOLID principles were as I named them off.

I have every reason to believe you when you say AA has bad programmers/contractors, but SOLID is just some random software process marketing buzzspeak and not knowing those particular "principles" isn't a useful indicator of anything other than not being exposed to that particular brand of process marketing buzzspeak.

FWIW, I'm not attack any of the ideas of SOLID here, just the idea that knowing of or not knowing of any specific brand of process marketing buzzspeak is a useful indication of anyone's skill.

Why would knowing the SOLID principles have any real relevance when the systems in question are probably written in COBOL and date back at least 4 decades?

> Only one of them knew what any of the SOLID principles were as I named them off.

Here's to hoping I don't ever end up where you are. Let's just let programmers do programming instead of brushing up on their MBA/"Six Sigma" style time wasting.

It is very useful to know about the lessons of those that came before so that you don't end up making the same mistakes. I'm not the OP, but if you don't care about theory at all, then I'm the one hoping I never end up where you are. I don't want to waste thoughts on trying to figure out the same damn problem that was solved 20 years ago. Yet, this is exactly what many programmers spend most of their time on.

Right, because principals related to OOP would clearly be useful to COBOL running legacy systems and remembering marketing acronyms are indicative of how much a programmer is going to waste time.

Most likely no. But the good news is everyone will assume they caused the problem.

If you board, you will be on the tarmac for a while:


"A while" -- no kidding!

UPDATE: 3:02 P.M.: The Federal Aviation Administration has advised that American Airlines has extended its ground stop to at least 7 p.m.

I'm picturing a disk failure, or a RAID5 with two dead drives. Someone at AA franticly running disk recovery software.

"Oh god please work ddrescue...."

Nice to see their legacy systems were kept up to date and maintained.

I guess this is part of the "maintenance"... :)

As a frequent US Airways flier, this makes me feel really warm and fuzzy about the upcoming merger with American Airlines.

I'd like to see a paper on American Airlines software architecture. Although I doubt it'll pop up on "High Scalability" any time soon. Do we know anything about Sabre's platform? I imagine its highly redundant..how could they have a nationwide outage?

On twitter, Sabre says they're not having any issues, and American Airlines retracted their statement pointing the finger at Sabre:

Clarification: The issue is w/ our ability to access our res system & not w/ @SabreNews. We apologize to Sabre & customers for confusion.


The hosting company that American Airlines uses is Sabre

This is the same company used by Virgin America, US Airways, JetBlue and Alaska Airlines

Also, a bit of trivia: AA invented Sabre. Sabre can actually trace itself back to AA's first attempts at using computerized business process automation in the 1950s, before it was spun off into a separate company in 2000.

Should anyone be bored enough for a read, I actually think SABRE's history is sort of interesting: http://en.wikipedia.org/wiki/Sabre_(computer_system)

A really interesting book I read about this is Hard Landing by Thomas Petzinger Jr. This contains more than just a history of Sabre. I read it more because I work in the travel industry ;-)


on the other hand Sabre's acquisition of Dunder Mifflin was a move in a questionable direction.


The reality is much more complex than that. There are numerous systems at AA for booking, reservation, ground ops, flight ops, airport integration, etc.... Many of these interact with SABRE, many of them are layered on top of each other but some only partially. Quite frankly it's pretty scary over there.

Isn't there an opportunity to consolidate what the IT needs really are with the US Airways merger?

"Integration costs" can be allocated to supporting what the business actually needs, and in the long-term there will be a cost saving. I appreciate the reality is not so straightforward.

US Airways uses SABRE as well, but let's just say your completely rational idea were to be adopted. The simple fact is they cannot execute. See my other comments.

SABRE is more of a platform than a hosting company. It even had it's own programming language at one point.

Yes, you're right. Hosting has a different meaning in the software industry - the correct term is 'Airline reservations system' but the colloquial term used in the travel industry is 'airline hosting'.

I got stuck for 2 hours waiting for "paperwork" flying from Chicago to Philly. At least I managed to finish my book.

It's more than just paperwork: one of the systems that was down calculates things like fuel loads and flight plans

I imagine that paperwork was on quotes because that's what one of the employees told him/her.

Why haven't carriers like AA been displaced by more competent (and passenger friendly) alternatives yet?

"How do you become a millionaire? Make a billion dollars and then buy an airline." -- Warren Buffett [http://www.fool.com/investing/general/2012/05/23/if-you-have...]

To add to that, Virgin America is a favorite airline yet struggles to make a profit


Thought that was a Branson quote:

"If you want to be a millionaire, start with a billion dollars and launch a new airline." -- Richard Branson

That sounds like something Richard Branson would say (I think I remember reading it in Business Stripped Bare), but I believe Buffett's quote predates Virgin Airlines entirely.

It's a highly competitive industry with very thin (read often negative) margins, which does not attract a lot of "normal" competitors. It's also highly regulated and extremely capital intensive, making it further unattractive to compete with the incumbents. On top of that, you have to get space at airports which, if it's full, you just can't do. My understanding is that most airports in major US centers are "full" during peak times (morning and evening rush hours). Some major international ones (LAX, JFK) are probably full most of the time.

All in all, it's a tough industry. That said, you get a few new entrants every decade. Frontier ('94), Jetblue ('2001?), Virgin America (2007).

I'm sure one of them has tried being nicer to customers, but next time those customers went to buy a ticket they bought one on AA cause it was $10 cheaper or left 30 min later.

Virgin has, but that has cost it horribly.

Someone did a focus group awhile back about all the nice things an airline could do. And everyone was really excited about all the niceties that were on offer.

But then they asked everyone to raise their hands and to pull them down once they were no longer willing to pay that much.

At around $30 extra, over half the people had put down their hands. At $50, all but one.

For whatever reason, people de facto hate paying trivial amounts extra even in exchange for significant improvements. That's why we end up with really shitty flight experiences.

I just booked a couple of flights from Europe to Asia, and it took a while to find the sort of low prices I've paid in previous years. It turned out that half the ticket price was for airport fees and taxes, so the airline was charging more and getting less.

Same here. Just bought tickets for SF -> NY -> London and back, and no matter how you slice it, you're paying a $500 "crossing the Atlantic" fee. SF -> NY hovers around $300-$350, but good luck finding a flight from NY -> UK for less than $1000, even though the flight is only 30% longer.

I've been planning a trip to Europe and flights to Heathrow and Barcelona from Tucson are < $1000 until Mid may. However, they all go through Atlanta or Houston instead of NY. Maybe it's a NY->UK premium? Strangely, the same flights direct from Houston or ATL to London are more than the ones from Tucson with layovers.

This is actually known collusion by the airlines that's tolerated on both sides of the Atlantic because the airlines basically make no money anyway.

Those "fuel surcharges" that add up to hundreds of dollars are imposed and collected by the airlines themselves. It's just another way for them to price their tickets.

Ya, make no mistake: most of those fees are just stealth price increases. I wish they would just come out with the real price directly.

Having been in the bowels of the beast I'm suspect of the quotes that the margins are thin. American and other traditional carriers (unlike Southwest) waste loads of cash maintaining a way of doing things that hails from the 1960's. I used to walk around imagining all the waste I would slash if I were CEO. It is truly appalling once you've seen it from the inside.

EDIT: Corrected spelling

Airlines are a pretty shitty business: commodity product, many competitors, low margins, high capital costs, high regulatory risk, high exposure to ever-rising oil prices, etc.

There's some areas in the higher end stuff, but right now JetBlue / Southwest are about as progressive as you get.

Airlines are a pretty shitty business: commodity product, many competitors, low margins, high capital costs, high regulatory risk, high exposure to ever-rising oil prices, etc.

I read an interesting blurb somewhere recently (forget where, sorry) that pointed out that it doesn't actually take that much capital to start and run a (small) airline. You can lease a plane and start out serving only one route, and you've got yourself an airline. Of course, how much capital is "a lot" is relative... let's just say that I won't be starting an airline anytime soon. But still, apparently it's easier to break into that business that you might think, looking at it from the outside.

Air Alaska is very innovative with using Satellite guidance for landings. It also has high margins compared with other airlines. http://www.nytimes.com/2013/03/03/business/alaska-airlines-f...

Well when someone else learns to do what AA does at the scale of AA then it could happen.

It's not like alot of smart people haven't' tried. The airline industry is littered with the corpses of failed companies.

Android anyone?

For a minute I had a '24' flashback and started packing canned goods and ammunition into a 'bug out bag'..

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact