

All American Airlines Flights Grounded, Experiencing Nationwide Computer Outage - jstreebin
http://techcrunch.com/2013/04/16/all-american-airlines-flights-grounded-experiencing-nationwide-computer-outage/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Techcrunch+%28TechCrunch%29

======
akcreek
I'm at DFW right now heading to LAX on AA. Computers just came up and they are
going to begin boarding shortly. Lots of unhappy travelers here as you can
imagine I'm sure.

Edit--

To follow up. It's been about an hour since the computers came back up in the
terminal and I'm finally in my seat on the plane. About 3 hours after it was
supposed to depart.

There were several canceled flights in my terminal and no other flights going
out so people had to stay overnight. Most were very mad as American was not
supplying food or hotel vouchers - all those people are coming out of pocket I
guess.

I heard quite a few saying things like, "No wonder they are going out of
business" and "I will never fly American again no matter what".

~~~
impendia
> American was not supplying food or hotel vouchers - all those people are
> coming out of pocket I guess.

Why are they not subject to a class action lawsuit? The problems rather
obviously seem to be American's fault.

Seems like some enterprising lawyer should buy a ticket, drive to the airport,
go to the AA gates, and start handing out business cards to everyone in sight.

~~~
bdunbar
> go to the AA gates

You have to be a ticketed passenger to get _to_ the gates.

~~~
w1ntermute
Then buy a ticket. It'd be a worthwhile expenditure.

~~~
joelrunyon
Great answer (but if the computers are down, can he buy a ticket?).

~~~
mikeash
Doesn't have to be a ticket on AA.

~~~
lisper
And it can be fully refundable.

------
ari_elle
"American said the issue was caused by an inability to get access to its
reservations system, called Sabre. The electronic system, often described as
the brains of an airline, is responsible for bookings and reservations but
also manages a wide variety of functions related to flights, including
printing boarding passes, online check-ins, ticketing, and tracking checked
bags. [...]

Sabre, meanwhile, said American’s system outage did not come from its own
computer systems. Other airlines, including Southwest Airlines and JetBlue,
use the reservation system and have not experienced any outages, said Nancy
St. Pierre, a spokeswoman for Sabre."

Source:

[http://www.nytimes.com/2013/04/17/business/american-
airlines...](http://www.nytimes.com/2013/04/17/business/american-airlines-
cancels-flights-after-outage.html)

~~~
ryanmarsh
SABRE was fine. Some of the systems at AA couldn't talk to it. Having seen
their systems none of this surprises me.

~~~
jchrisa
Somebody probably let an API key expire...

------
TallboyOne
I like how it says "apologize for ANY inconvenience" as if theres the
possibility that all planes being grounded making hundreds of people late for
meetings offers any room for there not to be an inconvenience.

~~~
jmathai
Or worse, parents with young children stuck at airports losing their mind.

~~~
pc86
Or worse than that, missing a meeting and being stuck with _other people's_
young children at airports.

~~~
jmathai
Unsure if you have kids but trust me. It's worse for the parents.

------
tpowell
I can't wrap my head around how extraordinarily complex airline
reservations/logistics systems are. I'm kind of surprised you don't see this
more often.

~~~
mimiflynn
Looks like they use Sabre:

UPDATE, 1:43 P.M.: The Federal Aviation Administration has confirmed the
ground stop. “GROUNDSTOP IMPLEMENTED BY ATCSCC ON AMERICAN AIRLINES NATIONWIDE
AND AMR AMERICAN EAGLE INTO DFW, LGA, AND ORD AT THE REQUEST OF AMERICAN OPS.
STOP REQUESTED AS RESULT OF AAL SABRE SYSTEM FAILURE.”

from [http://www.dallasnews.com/business/airline-
industry/20130416...](http://www.dallasnews.com/business/airline-
industry/20130416-all-american-airlines-flights-now-grounded-nationwide-till-
at-least-7-p.m..ece)

~~~
mjn
Looking at Sabre's own timeline, I cringe at what I suspect the codebase must
look like. From what I read between the lines here, it looks like neither a
carefully maintained legacy system, _nor_ a proper rewrite at any point along
the way, but more like 50 years of things bolted onto each other:
<http://www.sabre.com/home/about/sabre_history>

~~~
rbanffy
I have the utmost respect for systems that have been running for decades
without major rewrites, but having said that, I imagine there is a rock-solid
core that has, literally, withstood the test of time, surrounded by semi-
compatible systems that accreted around it since the mid-80's, with most
current systems only touching this peripheral mass rather than the core
services.

I've had contact with many systems that have their cores running on IBM
mainframes with peripheral systems running on .NET. I fear those.

~~~
ams6110
Transaction processing on IBM mainframes is typically very robust. It's what
they were built to do.

------
sterlingross
I am surprised no one has mentioned Rule 240. I have, more than once, gone up
to the agent when a flight was delayed and asked about this rule. They always
hesitate at first, but if you insist, they quickly make provisions to put you
on another flight.

This only works when it is the airlines fault and not an issue with weather.

Rule 240: <http://en.wikipedia.org/wiki/Rule_240>

------
nlh
Putting my tinfoil hat on for a moment...

I wonder if there was a threat against the airline? Perhaps something enough
to be credible but not so specific as to be pinpointable, and rather than
cause widespread panic by saying "ground every AA flight until we sort it out"
(which would freak people out beyond belief), someone figured out that pulling
the plug on their mainframe for a few hours would have the same result without
the panic. Just some pissed off passengers.

Or maybe I've just been watching too much Homeland. Sigh.

~~~
ryanmarsh
That's just your paranoia. I wish I could speak more freely than I already
have (see my other comments) but the fact is, crappy software, written by
crappy contractors all tied to a 53 year old system of record.

~~~
NewAccnt

      Old software + "wish I could say more" = conspiracy
    
      /joke
      /defensive mechanism

~~~
ryanmarsh
Comedy of errors... sometimes a very dark comedy...

------
tquai
Just a general comment on airlines:

With US-based carriers, expect low prices, inflating awards points/miles, and
bottom-level service.

With non-US based carriers, expect higher prices, no awards programs, and
excellent service.

This is my experience flying in N. America, Europe, and Asia.

~~~
deelowe
Huh... Explain how Southwest has done so well then? They are cheap,
profitable, and have the best service.

~~~
tquai
I compared US-based carriers with non-US based carriers. Southwest, a US-based
carrier, competes with other US-based carriers. You're right, Southwest is a
great choice if you're flying in the US, but that's not what my post is about.
You can't compare Southwest with Cathay Pacific or EVA Airlines because they
operate totally different routes.

Same thing regarding EasyJet comments, which is another "niche" airline. Try
flying EasyJet from LAX to NRT, for example.

------
dia80
I feel for the poor sods thrashing away trying to make this right. Do
developers in mission critical areas earn a premium? This could knock a couple
of years off your life in stress.

~~~
ryanmarsh
LOL, yah right. No they don't earn more. AA does not pay well. Also most of
the legacy systems are maintained by contractors.

~~~
runawaybottle
What do they earn?

~~~
ryanmarsh
I'm told, "well below market". I've talked to probably 40 programmers there.
Only one of them knew what any of the SOLID principles were as I named them
off.

~~~
clobber
> Only one of them knew what any of the SOLID principles were as I named them
> off.

Here's to hoping I don't ever end up where you are. Let's just let programmers
do programming instead of brushing up on their MBA/"Six Sigma" style time
wasting.

~~~
jmcqk6
It is very useful to know about the lessons of those that came before so that
you don't end up making the same mistakes. I'm not the OP, but if you don't
care about theory at all, then I'm the one hoping I never end up where you
are. I don't want to waste thoughts on trying to figure out the same damn
problem that was solved 20 years ago. Yet, this is exactly what many
programmers spend most of their time on.

~~~
clobber
Right, because principals related to OOP would clearly be useful to COBOL
running legacy systems and remembering marketing acronyms are indicative of
how much a programmer is going to waste time.

------
billiam
If you board, you will be on the tarmac for a while:

[http://www.dallasnews.com/business/airline-
industry/20130416...](http://www.dallasnews.com/business/airline-
industry/20130416-all-american-airlines-flights-now-grounded-nationwide-till-
at-least-7-p.m..ece)

~~~
uvdiv
"A while" -- no kidding!

 _UPDATE: 3:02 P.M.: The Federal Aviation Administration has advised that
American Airlines has extended its ground stop to at least 7 p.m._

------
nwh
I'm picturing a disk failure, or a RAID5 with two dead drives. Someone at AA
franticly running disk recovery software.

~~~
toomuchtodo
"Oh god please work ddrescue...."

------
edandersen
Nice to see their legacy systems were kept up to date and maintained.

~~~
umsm
I guess this is part of the "maintenance"... :)

------
drcoopster
As a frequent US Airways flier, this makes me feel really warm and fuzzy about
the upcoming merger with American Airlines.

------
capkutay
I'd like to see a paper on American Airlines software architecture. Although I
doubt it'll pop up on "High Scalability" any time soon. Do we know anything
about Sabre's platform? I imagine its highly redundant..how could they have a
nationwide outage?

~~~
dunham
On twitter, Sabre says they're not having any issues, and American Airlines
retracted their statement pointing the finger at Sabre:

 _Clarification: The issue is w/ our ability to access our res system & not w/
@SabreNews. We apologize to Sabre & customers for confusion._

<https://twitter.com/AmericanAir/status/324227267066736640>

------
eddmc
The hosting company that American Airlines uses is Sabre

This is the same company used by Virgin America, US Airways, JetBlue and
Alaska Airlines

~~~
sjm-lbm
Also, a bit of trivia: AA invented Sabre. Sabre can actually trace itself back
to AA's first attempts at using computerized business process automation in
the 1950s, before it was spun off into a separate company in 2000.

Should anyone be bored enough for a read, I actually think SABRE's history is
sort of interesting: <http://en.wikipedia.org/wiki/Sabre_(computer_system)>

~~~
eddmc
A really interesting book I read about this is Hard Landing by Thomas
Petzinger Jr. This contains more than just a history of Sabre. I read it more
because I work in the travel industry ;-)

[http://www.amazon.com/Hard-Landing-Contest-Profits-
Airlines/...](http://www.amazon.com/Hard-Landing-Contest-Profits-
Airlines/dp/0812928350)

------
meemo
Prescient article: [http://www.technologyreview.com/view/513391/why-a-botched-
it...](http://www.technologyreview.com/view/513391/why-a-botched-it-project-
will-destroy-a-major-corporation-in-the-near-future/)

------
maxdemarzi
I got stuck for 2 hours waiting for "paperwork" flying from Chicago to Philly.
At least I managed to finish my book.

~~~
jcdavis
It's more than just paperwork: one of the systems that was down calculates
things like fuel loads and flight plans

~~~
pyre
I imagine that paperwork was on quotes because that's what one of the
employees told him/her.

------
w1ntermute
Why haven't carriers like AA been displaced by more competent (and passenger
friendly) alternatives yet?

~~~
soupboy
"How do you become a millionaire? Make a billion dollars and then buy an
airline." -- Warren Buffett
[[http://www.fool.com/investing/general/2012/05/23/if-you-
have...](http://www.fool.com/investing/general/2012/05/23/if-you-have-to-
invest-in-airlines-dont.aspx)]

~~~
drpancake
Thought that was a Branson quote:

"If you want to be a millionaire, start with a billion dollars and launch a
new airline." -- Richard Branson

~~~
Encosia
That sounds like something Richard Branson would say (I think I remember
reading it in _Business Stripped Bare_ ), but I believe Buffett's quote
predates Virgin Airlines entirely.

------
monsterix
Android anyone?

------
SODaniel
For a minute I had a '24' flashback and started packing canned goods and
ammunition into a 'bug out bag'..

