I am affected by this. We got halfway into our flight only to find our next leg was cancelled. SWA will not (cannot?) rebook anyone until the 31st. Our return flight was going to be Sunday, so we rebooked from our halfway stop back home on Sunday.
Here are some crazy things I have encountered.
Rental cars in our city are sold out. Same for cities within two hours drive. The websites will accept reservations, but when you show up, they tell you they have no cars. Because it's the holidays, busses and trains are booked and any flights on other airlines are crazy expensive ($1500 one way). When you are stuck in a city, you are probably truly stuck there.
Your only hope of dealing with SWA is waiting in line at check-in or a gate. The phones don't work. Online chat doesn't work. The lines are long and slow.
Luggage is hit or miss. If your bag was pulled off a plane, you might find it in baggage claim, but most bags are on a plane or on the tarmac. SWA told us it may be 30 days until we get our luggage. They won't pull bags for people, and the agents that we spoke with acknowledged and felt for people who may have had medicine in them.
The workers are as befuddled as the passengers. They have been very nice and as helpful as they can be, but their phones haven't been working and their computer systems have been slow.
On Twitter, someone posted a video of the announcement at Houston Hobby about no flights until the 31st and people keeping their receipts for hotels, etc. They said the same thing at our airport.
People in the airport are so mad. It's unfortunate because it's not constructive. But tempers are flaring, and frustrated passengers who finally get to talk to an agent end up slowing things down because they spend a lot of time trying to hear something they're not going to hear.
Ultimately, this is an operations failure. Companies talk a lot about accountability, but the typical way you hold people accountable is by replacing them with more capable people. It will be very interesting to see if any executives leave SWA over this. If not, I would say that no one was held accountable.
To close, my family and I are fine. This is but a minor speed bump in life. No one is dying, and we will see how SWA takes care of the extra expenditures. Some people aren't so lucky. They have meds in bags, or finances that don't allow them to spend multiple nights in a hotel and get Uber trips for a few days. Hopefully, SWA takes care of them, too.
If there are no cars available near you, and you want to drive, try U-Haul or similar truck rental places. It’s probably not going to be a super comfortable ride, but you might get home.
That's a good idea, but not really feasible for a family. After the rental car fell through, we resigned ourselves to not getting to our destination and decided to make the most of the situation we have (explore a new city we have never experienced beyond an airport or a highway) and solidified our return flight.
the other 3 (expensive) hacks I can think of are charter a flight, and draft people stuck in the same situation to help defray costs, rent a 15-passenger van or bus to do it; buy any vehicle you can find and then resell it at the other end
For those in a situation you may want to consider renting a truck from Uhaul or similar. They’re only open during business hours and will be way more expensive but a vehicle is a vehicle.
I can’t say I recommend then offering to shove everyone else in the back.
This has been one of the reasons I heavily lean toward direct flights now - that way if stranded im either at home or my destination.
There used to be a website where you could post that you were driving from A to B and other people could opt in to go with you and hitch a seat. That would be great for stuff like this.
Friend of mine used something like this to get from LA to the bay area.
The car had 2 girls in it who hotboxed the car the entire 6 hour drive, then missed his exit near San Jose and got pulled over for an unrelated reason.
Says if he wasn't married now he would do it again in a heartbeat.
I used something like this to get from Berlin to Amsterdam. I wasn't hotboxed but did get to spend the ride chatting with an attractive fashion student. It was cool because I'm not involved in fashion at all but my uni was using that industry for a lot of examples in lessons.
My local uhaul charges a day rate and $1/mile, and gas is on you. Still you can get lucky if they need to move a truck your way. They'll practically give it to you.
Thanks for the update. I’m sure the people around you appreciate the calm you’re demonstrating. I hope you’re able to salvage some of your holiday plans.
I had a similar situation happen to me on my way back from a wedding. Got stuck in Charlotte with two small kids. It made me realize a bit more tangibly how absolutely terrible our public transportation is here in the US.
I don't know the cause of your specific situation. But with respect to 5400 cancelled flights, I'd be surprised to hear that there's a transportation system (public + private combined) that can handle a sudden and unexpected influx at a time where things are already at peak-capacity (holidays)
No, but in my particular situation (which predates the Southwest issue), had I been in Japan, Korea, a decent European, etc I could have managed to get a train ride from a city the size of Charlotte to a city the size of my hometown.
Me thinks you are quite right: back of the napkin math indicates the average flight carries 64.44 passengers. So just the Southwest flight cancellations stranded 347,976 passengers. That’s a lot of people to re-route.
Southwest boards three groups of up to 60 passengers each. Every Southwest flight I have taken post-pandemic has been full. I’d bump that average up to the 120-180 range for Southwest flights this time of year.
Yeah, but they're not all in one place. If the US had a genuinely functional public transport system, I doubt there would be much problem finding seats for all or the vast majority of the stranded people on trains going to or near their destinations.
There's 2 types of public transport: intra-city and inter-city. Intra-city doesn't serve the same use case as flying, and can't generally be repurposed to inter-city transport, so it should be discounted. If the US had inter-city rail that people actually used, then it would almost certainly be near capacity in a normal holiday season. Unless you assume there are a lot of extra train cars that aren't normally being used but could be taken out of storage, there's no way that it could absorb such a large influx of stranded passengers on short notice. Inter-city rail generally operates like airplanes in that only one ticket can be sold for each seat - there's no standing room on a multi-hour train ride.
> They won't pull bags for people, and the agents that we spoke with acknowledged and felt for people who may have had medicine in them.
Casual reminder to keep all medicine in your carry-on! Even gate-checked bags get delayed and lost. I never realized that could happen with a gate-check until it happened to me. They loaded all of the gate-checked bags onto our plane, but then that put it over a weight limit, so they took a pallet or two of them off and put them on another flight. Of course, it was a complicated multi-leg international flight to boot.
Anybody who puts important meds into checked luggage is really to blame for their problems. Bags get lost even when airlines don't have catastrophic breakdowns of their normal operations and taking this risk is really not necessary as meds can totally be carried in carry-on. It's really unfortunate that this is causing problems for people but it's totally preventable.
Your getting downvoted a lot - but as someone who has been racking up north of 100k miles/year for five years pre-pandemic (somewhat less now) - anytime I checked luggage I essentially needed to be mentally and emotionally prepared to say goodbye to it. I booked my tickets with Chase Sapphire which has very good missing luggage/toiletries/etc... insurance - and had my luggage go missing five times in as many years - though, I will say, all five times I had the airline deliver it to my destination hotel within 48 hours.
Regardless - don't check anything you aren't prepared to say goodbye to.
Yes, but I think the comment is being downvoted for tone, not accuracy. It’s not nice to lecture people when they’re down, even if they’ll never read it.
You shouldn't be downvoted for this. Southwest (and other airlines) specifically recommend bringing all medication in carry-on bags. I have had checked bags delayed multiple days on other airlines. It happens.
There being a possibility of a negative event providing motivation for a strategy to mitigate the event doesn't provide justification for realizing the negative event through sheer incompetence nor does advertising your incompetence ahead of time clear you of the blame for said incompetence.
Except that even if you only bring a carry on, it’s not guaranteed that you won’t be hit with the “we’ll have to check this bag free of charge for you”. I’ve been in my seat, the lady takes my bag down from
The storage to fit something else, can’t fit my bag back, and decides to check my bag without even talking to me. Other times it happens as you walk down the aisle and find no room, it’s very common.
Meh, it's avoidable. I travel with a medium-big backpack as my only carry on, usually, and have never been gate-checked. The roller bags which barely fit in the overheads and have no compressibility are the ones that get gate checked.
So you travel for a meeting for a day and back? Ever travelled from warm weather to cold weather? Ever travelled to a place where you’ll need two types of shoes? Or multi city with different weathers, different engagements… I don’t know how you can believe that “you can just travel with a small backpack anywhere!” “If I did it, it must work for everyone!” “99% of the people have carry ons bigger than a backpack because they are stupid and literally do it for no reason”. Is that the logic you’re following? I don’t get it.
But also I've traveled for a month (us to India) from my backpack. One week of clothes, a laptop, some cables, notebook and pens, perhaps a couple more books, a few toiletries. And still some room for one 'fun' thing, like a small synthesizer to jam on.
It's a bit like ultralight backpacking; you can cut down quite a lot one you get in the mindset. I definitely don't claim it works for everyone... But probably most people can travel rather lighter than they normally do.
Fwiw, my travel backpack is pretty similar to this one. Definitely a bit bigger than the standard backpack, much smaller than a hiking backpack, and rather smaller than a roller bag.
100% agree that happens, and frequently, but it's also perfectly avoidable - don't try to maximize volume. Bring a carry on that fits underneath a seat. I typically travel with a rectangular backpack (plus, if needed, a check in, which I acknowledge may not make it there). It gives one an independence and self sufficiency that's completely worth it.
Still there are plenty of medicine you don’t want to miss for a few days. For me and my family anything that we absolutely need travels in hand luggage always. In my experience getting a lost bag can be a huge pain specially if it was lost during a layover
This is really harsh but it’s good advice. I’ve never checked important medicine but I never really thought about the danger of losing access to it in a random city. Fortunately I think local pharmacies would be pretty understanding in this situation… and give people temporary access to missing medicine… hopefully… perhaps the airport could even act as a middleman (I got urgent care drugs at a foreign airport once when I was sick).
Pharmacies cannot and will not do that (except for insulin, for which it's now allowed in some states, and maybe a few other non-abusable things.) If it's a controlled substance -- which includes most things you'd be worried about withdrawal from -- forget it. If it's a more tightly scheduled drug, you'll have trouble even getting an unfamiliar doctor to prescribe it. It's not that they don't care about your withdrawal syndrome, but they generally care much more about not losing their license.
I think it would be a matter of calling the prescribing doctor’s office, explaining the situation and giving them the details of the pharmacy you’d like to pick up the prescription from. The biggest hurdle would probably be insurance, who doesn’t want to pay for more medicine than was necessary.
Good luck with that Dec 24-26 (the time frame we're discussing). You might get an answering service that can get a message through to your doctor but the office is likely to be closed and empty. Then you still have to find an open pharmacy with your medicine in stock.
I don’t know that pharmacies are allowed to do that. What you usually have to do is call your primary care or other doctor and explain the situation and have them send a prescription to the new city
This is why corporations can't be allowed to just do whatever they want. A rental company renting non-existent cars to you is a breach of their business purpose, to sell a product or service. You've spent time and money to get nothing, against what you were sold.
Who can step in to make some baseline rules that corporations have to abide by so people aren't screwed like this?
You could find flights earlier for around $750 to some destinations via private jet charter (subscription) company some friends did that while having similar problems.
No excuse, none? Not even the harried passengers at the terminal suddenly being told they need to check their carry-ons due to lack of overhead bin space, right before they're about to board?
You have never forgotten to pack something, or perhaps forgotten an important detail in a stressful situation?
I had an international flight every month on average in 2022, across Europe, and I was never explicitly informed about this by airline or any of the friends and coworkers.
As I understand it, Southwest is particularly badly affected by such issues because they run a lot of flights that are not in a hub-spoke model, but rather serial flights one non-hub city to the next and next (like eventually coming back in a loop). You can see this by going to Flightaware.com for example, and following back a flight's previous destinations. See for example https://flightaware.com/live/flight/SWA1092/history/20221223... and "track inbound plane" a couple times.
They jump around the country, much less frequently going back to a hub as other airlines do. That means that the planes and crews have a relatively harder time recovering from system-wide disasters because they don't have as part of normal operations as much ability to centralize or pool resources and get people/planes reorganized. (everyone go back to base, consolidate passengers, crew, planes and redeploy them and sort things out in one place)
Unfortunate, but that's their model. Good for some purposes, not so good for others. Maybe it's them being quirky and an active choice. I mean, up until a few years ago they did not fly to Hawaii because their scheduling system / people / processes did not want to have redeye flights.
They've operated without the hub-and-spoke model years but they haven't had to operate with over 8% of their staff leaving. They're understaffed. It was a major issue with Southwest all throughout 2022 and it got brought up on their earnings call with investors. They're a budget airline and they can't afford to take that kind of staffing hit.
From a friend that works there this is not a staffing issue. It is a software issue. Apparently, their system that tells them what staff is where is broken and doesn't know where anyone is. So now they need everyone to call and tell them where they are to update the system, but the phone lines to do so are totally overwhelmed. This is also causing them to be unable to book hotels for the staff that is stranded. Most people are just buying their own hotels and hoping for reimbursement in the future.
This is interesting and topical considering what's happening in places like Twitter..
..And the one thing that might have saved Southwest in the past, its ferocious employee loyalty and willingness to go several extra miles when the shit hit the fan because they knew they were being taken care of, has been utterly destroyed by the new management.
People were loyal to Herb (not SWA) because Herb was loyal to them. Herb’s gone, loyalty’s gone.
You could both be right -- could be that a hub/spoke model would fare better in a software failure, because everyone would return to the hub and things would be straightened out the old-fashioned way.
Not having staff didn’t seem to stop them overselling what they can handle, however. They took a risk on stretching as far as they could go in an ideal environment and here we are.
This is probably tied to how deeply ingrained the overselling habit is in the airline industry in general. They're legally protected when doing this, and it's why that doctor got dragged off that flight.
Proof appears to be in the pudding here. They absolutely overbook, all airlines do. Dated a few attendants over the years and they’ve all echoed similar experiences with folks being double booked especially so during this time of year. I’ve been bumped on a SW flight, got the extra travel voucher so it worked out. They absolutely do overbook.
If you read the linked faq, there's a difference betwen overbooking and overselling. If Southwest sells all the seats for a segment, but they end up flying the segment with a different plane than scheduled, there may be less seats than scheduled passengers. That's more likely to happen furong the holidays when flights are very full and weather delays are common and there's more equipment changes.
The difference is intent. Overbooking intentionally creates conflict likely to result in a ticketed passenger unable to board. Overselling also results in a ticketed passenger unable to board, but was not intentional; at least not directly intentional. You could argue flying planes with different capabilities offers the possibility to have a lesser plane subsituted and that's an intentional choice, but...
Another way to think of it is what could an airline have reasonably done to avoid the situation? If it's overbooking, the reasonable thing is to not overbook. If it's overselling, they could choose not to fully book their scheduled equipment, but is that reasonable? They can't choose to have 100% reliable planes and crews and weather and ground operations. Stuff happens, and it's certainly reasonable to be upset when it does, but understanding why it happened can be helpful, so making a distinction between overbooking and overselling makes sense to me.
The airline sold you something that they can't deliver because they refuse to keep extra planes around. They refuse to keep extra planes around because that would eat into profits, and would mean their execs wouldn't be able to buy their third gold-plated yachts.
No, they're all 737s, but there's a lot of variation within that.
Seatguru says [1] southwest flies three variants, 737-700 with 143 seats and 737-800 and 737-Max 8 both with 175 seats.
If a -700 gets substituted in, that's a lot of missing seats. I've also flown on planes where one seat is out of service for whatever reason and usually has a plastic cover on it.
As a policy Southwest is one of (the?) only airline that doesn’t overbook flights. They only sell the seats they have. From a business perspective it would be dumb to not offer all those seats for sale.
This is definitely their fault, but nothing like United pulling the doctor off the plane
That is definitely not the case. The real issue is that even the staff that did show up didn't know where to go. There were employees lost for hours at Denver because the call-in scheduling system went down. Some employees hit their limits for work time before they could even obtain their assignments.
That’s a symptom. The root cause was lack of resiliency in airplane routing.
It’s the only thing that explains why Southwest was uniquely impacted this week. Every other airline has staff issues, and the same weather to deal with.
With a point to point network, the coordination problem is much larger than with a hub and spoke network. That is probably what pushed Southwest’s software over the limit. But remember, the software didn’t fail by itself in a vacuum. Resiliency problems during a weather event were the root cause.
I don’t fly much but I noticed things like “the same flight number takes off at the same time each day and is always the same plane as a different flight number coming the other direction.”
It must really help all the employees with routine and consistency even if it’s not optimal.
Most airlines have schedules that are consistent day-to-day. It's the efficiency vs. resiliency tradeoff that's interesting. I'd probably summarize it as "don't fly Southwest in the winter."
That said, I flew Southwest from SJC to LAS for CES one year, connecting in SAN. Weather wasn't great, and they'd put you on the next available flight with an empty seat. They were even able to shuffle people without going up to the podium. Legacy carriers would have drug their feet, there'd be a line, and they'd charge for the privilege of changing flights.
I'd look at it the other way around: cancelling is so annoying for them that they're often the last ones to do it (barring catastrophic collapse, of course).
When I was traveling weekly out of Chicago, I always made sure to bring my Southwest credit card, just in case. Southwest sucks, but it gets you home.
My sister was recently stranded in DFW trying to make it to SFO when American Airlines canceled her flight. They were happy to substitute another flight... to Sacramento.
So the two-hour round trip to pick her up from SFO turned into an eight-hour round trip to Sacramento. I'm amazed this was considered an acceptable substitute. Would have been nice if American was willing to get you home in the event of canceled flights.
(We could see available seats on flights from DFW to SFO at that time from Delta and Alaska. But those seats were "not available to American rebooking agents". It seems like that should have been American's problem, not ours.)
Footnote: even though my sister had to be rebooked onto the American flight to Sacramento, American didn't bother rerouting her luggage, which they sent to SFO. I guess canceling the flight meant "the plane will still fly, but without passengers".
You live 1 hour away from SFO but 4 hours away from SAC? SFO is two hours away from SAC. How is that possible? Maybe it's 3 hours away if there is traffic.
I realize that this was inconvenient and a big hassle, so I don't want to make light of it, but something seems off with the geography.
No, the travel time to and from SMF doesn't total eight hours. But there is overhead involved in making long trips that isn't necessary for short trips.
- You need to allow for a wider margin of error in predicting travel times, which means leaving earlier than strictly necessary. But leaving early doesn't mean getting back early; you have to wait for the plane to debark.
- This trip was long enough to be more demanding than an electric car could handle, requiring 30 minutes of time spent parked and charging the car.
- More time was wasted waiting for the luggage to show up; we were not informed that it hadn't been sent.
At the very least, they were able to get her over the Sierras which are being dumped on right now. If they got her even to Reno, good luck getting over Donner or Echo summit for the next few days.
United proactively rebooks you on a new flight if you will misconnect, gives you options in the app, and issues waivers that let you avoid weather by rebooking your own flight (often waiving fare differences as well). Haven't flown Delta or American as much, but at least United's tech is a bit more modern.
As a matter of course I avoid United, and American doesn't have much presence out of SFO or OAK. Delta has done right by me each and every time something's come up.
Stuck on the tarmac in the snow and miss the connecting flight (last of the day)? They automatically rebooked me on another airline and I got a notification from the app as soon as I had reception.
3+ SFO bound flights delayed at JFK because of crazy winds at SFO? They proactively encouraged people to rebook, gratis. I rebooked on a flight the next day on nicer equipment, through the app, went into town, grabbed some bagels and had a flight at a nearly ideal time of day.
Regional plane goes tech? They had a red coat out and about keeping everyone informed.
Missed a flight because I misread the departure time? They sold me a same day ticket on the next flight at a hefty discount and I kept the inbound leg.
Southwest has a huge disadvantage and it's not tech: it's the lack of interline agreements. Nearly every other airline (except perhaps Spirit) can rebook you on another carrier when things go sideways. Southwest simply can't. With an interline agreement in place you'd have far fewer people getting stuck with exorbitant last minute fares.
In general though don't fly when you're getting unusual weather. Less than an inch of snow at PDX throws everything into chaos (and Portlanders call it fucking snowpocalypse). A few inches of snow at PWM and they don't even blink.
Notably, per the Alaska agent I spoke to about my cancelled Alaska flight, Delta doesn't let other airlines rebook on Delta flights. That may just be an Alaska/Delta thing but it's not obvious full reciprocity.
I find it hard to believe that Delta (or any other major airline) wouldn't reciprocate. When I got rebooked onto an Alaska flight, the gate agents were openly annoyed at having to accommodate Delta passengers. However, the relationship between Alaska and Delta has grown more adversarial so who knows.
I should also add that a few years back when my Southwest flight out of SFO got cancelled I was able to book a seat on a flight out of OAK with a minimum of effort. I think I had to pay the difference in fare though. When it works, it works, but the go it alone attitude will only take you so far.
Someone below linked this company as an example of resiliency tools used in airfare. Interestingly, there's a testimonial from United on one of the front page videos.
Maybe not 1:1 for what you're describing, but it solution/reason, but does seem like a possible sign that they're investing in proactive tools.
Yeah delta will do this as well. The app will let you pick any flight that day for free if you do not like the one it auto rebooked you on. I usually just take the one it gives me, but if it tries to route me through DTW or MSP with bad weather in the winter I will try to find one that goes through ATL instead.
My American flight on Monday was delayed (crew rest requirement was the reason given, which actually means not enough crew). It was a connecting flight so once it was about 90 min delayed it wouldn’t be possible to make the other leg.
AA live chat said they couldn’t help (other than rebooking 2 days later which would be pointless) but when I called they were able to drop the connection inbound and outbound so I could drive to the hub city airport (cost me like 2 hours each way, nbd).
They fixed it in like 3 minutes on the phone but it seems like they have to do it in a super hacky way because I had to check in at the airport which was no fun with long holiday lines.
Delta’s irregular ops procedures have always treated me right (in comms and in performance) and their app is pretty decent.
If I’ve had a missed connection en route, I’ll generally land and the app tells me what they’ve already rebooked me onto, but I have a choice of many different alternatives (usually) and can pick among them without cost and without waiting in a call queue.
Delta offers you a buggy website to rebook for no fee. I changed to a flight that was about $1,000 more expensive a few days ago. (note that the significant price change is an edge case caused by travelling on Christmas being particularly undesirable)
United is absolute worst. Many of my colleagues are frequent flyers on United. I just do not trust United. They lost it completely when they caused that unnecessary incident where they forcibly removed that doctor from a plane. And then there is United Breaks Guitar, the viral book about their "legendary" customer service. I will never fly that airline.
In that book, the ordinary logistical setup is that ships tend to cover individual transit routes, which means that a delay affecting one ship doesn't spread through the system. Malcom McLean tries to set up a system of ships that always sail east instead, and it fails very badly, because delays on each individual leg of the (infinitely long!) route accumulate instead of happening and then fading away.
Optimal isn't the right way to think about it. It's a tradeoff. Hub-and-spoke is usually better at getting you to your destination in less absolute time given the same number of total flights since you can have more frequent "shuttle flights" that travel to the hub, exchange goods/passengers, and shuttle back. Point-to-point on the other hand, is better for minimizing travel time since you go directly to the destination.
I heard it’s optimal for staffing though because you can just have people on standby living near your spokes, which would be important if you’re dealing with cancellations, which can severely cascade with point to point
Yeah that model is likely why when I schedule direct flights months in advance I get my flight rescheduled for multi stop flights, often with absurdly short layovers of 15 minutes (who is going to mark that flight?).
It was a pain so much I stopped flying them. I’d buy a ticket and have to babysit it so that a flight from noon to 6pm didn’t morph over several changes into a multi stop marathon from 8am to 9pm…..
I wonder if this event will end up being a demonstration that they aren't sophisticated enough to use their operational model. I would think planning decisions would at least try to account for disruptions and recovery time.
I see lots of people who are at least quite a bit less likely to use them in the future (and they are still in the middle of trying to fix it).
Both can be true. Some people will swear them off. Some won't. Some SWA will win back with steep discounting.
For me (I am affected), this is actually another in a series of recent events that are making me reconsider my preference for SWA. They are no longer a "cheap" airline, routinely more expensive than the other major carriers. Their planes are not nice anymore. I've flown on a few other airlines over the past few years and found their planes to be nicer with more features (like chargers and phone/tablet holders). And now this. The cancellations are one thing, but they totally botched the communication of it, and their practice of delaying flights throughout the day only to cancel half of them after several hours left people stranded.
Will I stop using them? We'll see how they respond, but they may not be my first choice anymore.
I’ll be driving 18 hours rather than risk a flight rescheduled 5 days (soonest available) after the original flight being cancelled, being cancelled again. Stuff sucks man. $2200 to rent a minivan for two days one way.
I feel bad for you/anyone having to pay extortionate rental car one-way rates because they're so variable/fickle. In future try something like below, before biting the bullet and paying full. But maybe in this circumstance with demand being what it is, there's no way around it.
For the ~75,000 travelers directly impacted, it will probably have some long-term impact in their purchasing decisions.
But for the 329,925,000 other Americans, many of whom have a long history and belief in Southwest’s reputation for customer service and fair policies? They will have forgotten by next week.
Yeah airline travelers are price conscious and it is a race to the bottom. If they offer some crazy sale or cheaper fares people will book it. Just look at Frontier/Spirit. They consistently get horrible reviews but people deal with it for a $50 flight.
People often try to say “airline travelers are price conscious” as if there are several options in the same price range and travelers will accept any reduction of quality or service to save a nickel (I’m not claiming you’re suggesting this). But in my experience with US domestic flights the options are basically one “cheap” decently tolerable itinerary, a few slightly cheaper itineraries that are like twice as long in total duration, and then a couple of slightly better itineraries with better amenities that literally cost like twice as much or more.
I just laugh at the upset attempts when you go to check in online: “get priority boarding and 2 inches of legroom for only an extra 50% on top of the ticket price.” I really don’t see much evidence that there was actually a race to the bottom. And I certainly won’t blame consumer preferences when I don’t see any options for slightly better service for slightly more money.
Boston is such a weird airport I'm not sure it's worth bringing up except as perhaps an exception that proves the rule.
BOS has the "advantage" of serving a fairly large population while also not being big enough to be a real hub for anyone[0], while being simultaneously big enough to have service from nearly everyone.
Unlike a lot of airports smaller or serving fewer people than BOS (and some of comparable size), you can get from BOS to a whole mess of hubs.
A few select routes (BOS to SFO as noted) are incredibly well-served because of the volume of lucrative business travel between the two and the fact that a whole mess of airlines already serve both airports.
[0] No, JetBlue doesn't count. Boston is as much a hub for them as CLE[1] was for Continental. I.e. a second class hub at best.
[1] CLE by comparison only really serves Cleveland. Columbus, Dayton, Cinci, Indy, Pittsburgh and probably a few others from a similar radius BOS draws from all have decent(ish) airports. All of those have basically the same problem as CLE or are worse in some way. I've flown through or into and out of all of them.
Just as with ISPs, for many people in the US, true airline choice is not a luxury they have. Depending on their origin and destination, there may be only one airline that flies it, or only one that flies without a ridiculous set of stops or layovers. Even if people want to switch airlines, unless they live near a major airport or have high flexibility on when and where to fly, it's not really practical.
Even when there are choices, the price difference between the choices can be absurdly high. You can't call people "cheap" when they choose the cheapest option, and the other options are 2 or 3 times the cost for service that's only slightly better, maybe.
Yeah, I have 2 reasonably drivable airports that are both served by Delta. It's even the case that I can mostly get a less expensive flight with a good itinerary (airport to hub to destination) or a more expensive flight with a bad itinerary (airport to hub to other hub to destination).
Is Southwest the lone primary carrier for many of their airports?
It also might be fine if they only have to deal with this kind of event once every few years but it lowers their costs substantially the rest of the time. I wouldn't love it as a customer, but who knows.
point to point was how every airline operated before gas prices and decreasing ticket prices caused airlines to focus on concentrating pax loads. it's not quirky; it's just not cost effective.
SWA can do this bc they operate a single aircraft type (737), have lower opex (they operate closer to spirit than delta internally and they do things like not allowing full GDS access to force leisure travelers to book through their website), and they keep their aircraft around a while. they also have a smaller network than the bigger airlines do, which further lowers opex.
However, SWA isn't truly point to point seeing as how a lot of their traffic flows through Chicago (MDW), Dallas (DAL) and Houston (HOU) and they have huge hangars and service ops out of these locations.
hubs aren't immune to huge cancellation numbers like this. American and United were heavily impacted during the 2021 Winter storm. Had the storm happened during the busiest peak travel season of the year like it did this year, they would have had record cancellations as well.
Delta had a similar outage due to a datacenter fire, grounding all domestic flights. Southwest was uniquely slow in taking days to start up again. And if the way my American Airlines ticket switched my birthdate to January 1st, 2000 is any indication, many airlines still need to modernize.
Most of the travel industry runs on old software that would horrify a lot of people here, especially those who've never worked for a large, 30+ year old company. When I used to interview a lot of people I made it a point to mention some of the more "interesting" aspects so they'd know what they were getting into.
One example: ever tried to book a flight a year in advance? On a lot (almost all?) of systems you can't, because the underlying date format is "DEC27".
Edit to address a couple comments: logistics are hard and there are plenty of reasons why airlines wouldn't want to support booking that far out. However, the reason you can book a flight 330 days from now but not 360 days from now is almost certainly due to the date format. (I believe the windows used are less than 365 days because it's helpful to be able to have dates in the recent past. I remember seeing documentation for 360, but AA and United seem to be in the 330-340 range on their websites).
As a fun side thing, I am also a travel agent with access to some of these internal systems on the booking side. The technology is incredibly antiquated. Most of the US runs on a system called SABRE, which is basically a MS-DOS system with a text command line interface and its own language. It's all ASCII text based (and all in uppercase). It's straight out of the 80s. Travel agents need to buy special "errors" insurance to cover any losses caused by fucking it up (a typo could accidentally cancel a ticket and cause the client thousands in losses rebooking it).
They actually have a GUI interface over it now with the ability for power/legacy users to drop into the raw shell, if they wish. From feedback, many of the older agents actually prefer the command line, because it’s muscle memory and an experienced agent can perform routine tasks that would take multiple screens in the UI with one hand in the way we’re comfortable with our text editors.
Granted, the rollout across airlines is probably glacial
I don’t blame them. Modern UX has a huge problem with something as simple as date pickers. Preferring you scroll through 90+ items when a simple textbox would suffice.
That is a tremendously fun fact! The little background things that keep society running. May I never be cursed enough I would ever have to directly work with such a system.
It'll probably never go away, but just be layered over like civilizations. Eventually our software is probably going to get so complicated that we just build new software to interact with old software to avoid ever fully shutting it down. Like building a fresh highway on the oregon trail
I remember using some version of SABRE through CompuServe back in the day. All command line stuff over a dial-up modem, but it was novel and cool to be able to book your own flights with it. It would be very annoying to still be stuck on that interface, though.
1) Replacing any software for an airline carries huge risk. They are barely operating ok with the software they have and holding it together with duct tape. Even something you might regard as ancillary, like a baggage handling software system, or flight catering software system, if it goes down, has the potential to disrupt thousands of planes and hundreds of thousands of passengers for days.
It is so significant an issue (to try to change some software, and just one out of many systems that have to talk to each other) that if an airline ever considers doing this, they may actually stop operations for some number of days while they do it rather than risk having operations go wrong. There are some rare examples of airlines doing this to try to change their systems.
2) Related to the above, airline management hates to be embarrassed by something that might work but has the potential to go badly wrong. So they are very conservative when it comes to replacing systems that are working, even if it's painful / much less functional than what they might achieve by a change.
Combine these factors (and many others) and it means that sometimes starting a new airline is simpler than trying to fix an old one...
Sabre has a 10-year deal with GCP to do just that. It's going to take a lot more than 10 years to get the thing off System/360 running in a bunker under Omaha airport though.
The typical answer for old behemoths: it was built because it was necessary to build it, and it won't change until a change is necessary too. Wanting that change is not enough, it has to become an almost mechanical constraint, and usually the constraint gets noticed when it far outweighs the costs (and not just a little). Or is a noticeable threat to the system's existence.
GDS — there’s really only 3 centralized stores of real-time flight/hotel/booking information in the entire world (Sabre/SABRE, Amadeus, and Travelport). Almost every American airline uses Sabre (American Airlines is an interesting case in that it does not technically in a legal sense, but actually it spun off and sold Sabre in 2000, so a lot of their core systems are forks of each other)
Complexity — Fundamentally you’re looking at a logistics software, except unlike packages you’re dealing with people who aside from expected destinations have travel lengths and time-in-air calculated down to the minute. Also unlike a package, a surprise multi-day trip, unexpected multi-leg journey, one day delay is not something passengers (and crew members) will accept or be at all ok with. And if any one thing goes wrong there’s going to be cascading failures down the line— so much that it may break your company’s entire operating workflow (e.g. Southwest) entirely, and no software can overcome that kind of organizational gap.
Airlines - There’s not many commercial passenger airlines left in the US, especially that fly nationwide. Good luck trying to convince one of these giant behemoths to move to a non-battle-tested system for core operations, especially when decades-old industry software and practices around that software exist.
Entrenched - Sabre is entrenched in airlines around the world. They don’t just provide the booking services, they do the flight tracking, the ticket handling, the upgrading, the in-flight upgrades, missed connection handling, the flight scheduling algorithms, the pricing algorithms, the pilot and flight attendant time tracking, ground crew management, even the terminal software at each gate. To replace SABRE, you would physically need to rip out and then replace software around the world. And because agents don’t work from an office usually, but at the airport, you’re going to need to conduct trainings and provide support around the entire service area, which for the largest airlines is the entire world
Scale — A lot of Sabre’s revenue comes from passengers boarded. It depends on the airline, but I believe the average is that each airline pays 10cents/customer boarded with their software (though with increases in passenger volume each year, it may be less now). Because Sabre is so prevalent, and so many flights use them, they can afford such a price. A company servicing just one regional passenger airline would absolutely not be able to compete on price, at least starting out
Also— Sabre’s software itself is actually reliable! As a corporation it is slow clunky and bureaucratic, but the actual functionality it provides is stable, battle-tested, can handle any travel edge case you can think of, and fast and efficient for those who know how to use it, while also good enough at day to day operations that it doesn’t take too much time to train new agents on how to use it for routine tasks.
SABRE is ancient technology, but very reliable and at the same time extremely inflexible. Last time I saw it upclose in the early 2000's much of the core was still coded in IBM assembler, although over the decades more pieces were slowly being modernized so I have no idea where it is now. Sabre is a horrifically un- imaginative company where projects are measured in years and not much every changes.
I think though Southwest's issues are more on their side.
Yeah building a new GDS today is an exercise in insanity, it's a huge complexity nightmare and switching probably impossible. I always wondered if AI could eventually improve things, but the existing GDSs are unlikely to care much to try. It's basically a (tri)monopoly you can never break.
Great write-up. This applies to other industries such as dealerships with parts and service and various purchase plans. The software sucks for everyone except finance/accounting and that office is beside the Pres's office and therefore it won't change.
"What are the reasons preventing flight booking software modernization?"
Let's ask the opposite question:
Why do you feel that software necessarily needs to be modernized ?
I'm not sure how well SABRE works but I do know how fast and efficient keystroking through a non-GUI interface can be and I don't know why expert mode interfaces should ever be replaced by unsophisticated mousey-mouse-mouse ones.
In context, it seems like modernization simply means that the software knows whether a flight was canceled and therefore can automate the status updates for the airplane crew. Nothing to do directly with UX.
The existing flight booking software that they use (Sabre) can and does do that, but Southwest’s issue is likely their insistence on using a homegrown management system on top of that. Southwest only switched to Sabre in 2021, so it’s likely still being implemented, and their homegrown approach has likely evolved with them from their founding, so it’s not something the company itself is likely organizationally prepared for.
At least, I feel confident in that analysis, since this exact same issue happened to Southwest in 2016, before they were using Sabre. Which would point to a chronic organizational failure
airline margins are paper thin these days. they weren't back when booking engines and reservation systems were being built. (SABRE was built by and spun off from American Airlines. Galileo was built by United before they merged with Continental.) this plus the absolutely insane business logic that goes into booking engines has made the effort extremely risky.
To be fair, I think allowing flights a year in advance is probably far more complicated than just updating the underlying date format. Even if they were able to solve that problem, airlines probably can't easily operationally plan that far ahead due to so many moving parts, i.e. committing to routes and schedules, planning for staffing that far ahead of time, ever changing government restrictions, fuel price fluctuations, inflation, geopolitical realities, staffing, etc. I mean, imagine if they did, and something like COVID comes along again, it would cause far, far more disruption if they had booked out the next few years in advance (we had no idea how long COVID restrictions would last while we were in the heat of it, it's only clearer now in retrospect).
Also speaking as a software engineer myself, it's almost never just a software fix that will magically solve everybody else's problems, that always ends up being just wishful thinking
While this is humorous in that there are limiting assumptions like this baked into the system, I also have to wonder, who needs or even wants to book a flight a year in advance? I dread planning out a flight 4 months in advance and dealing with the almost inevitable cascade of conflicts this introduces of juggling and rescheduling things to make things align correctly. One year makes me cringe.
There are events that I can see purchasing flights well in advance for. I used to a go to a conference that was held every other year at the same time, it would have been easy to buy tickets more than a year in advance for that without much concern. Eclipses, certain sporting events, or reservations for activities with a wait list of more than a year could qualify as well.
Despite that I am like you and rarely have tickets far in advance of a trip.
- Annual conferences or conferences that occur every-other year
- Planning family reunions because you need that kind of cat-herding lead time when you have 9 uncles/aunts on just ONE side of the family
- Periods where I have some spare cash I'd like to lock in a getaway with before I spend it or something unexpected like the invasion of the Ukraine drives up fuel costs and overall prices... or a global pandemic hits - would be sweet if I could have rebooked some of my trips for for 1-2 years out when the pandemic hit
- Travel for future medical stuff; at one point for 2-3 years I was taking my mom to the Cleveland Clinic every 4 months for periodic checks and it would have been super nice to be able to just book that stuff way in advance and have it all taken care of
Etc
etc
etc
I'd bet quite a few people would appreciate that ability
My family plans the yearly family get-together at the yearly family get-together. A year in advance. Except sometimes due to scheduling deconfliction, it's actually 10 or 11 months in advance.. or 13 or 14 months in advance. The exact date floats and sometimes we are planning trips more than a year in advance.
I could see it for major holidays. I spent too much money to fly home this year because I am bad at scheduling. I would consider booking next year's flight during this year's trip just so I know it's knocked out and I don't have to worry about it.
,, Most of the travel industry runs on old software that would horrify a lot of people here''
If you can see how it works, it horrifies me even more as a traveller, as from outside it just doesn't work a lot of the time.
Also if you just look at the video, we all know how bad these systems are, but are not able to do anything (starting anything new in the airline industry has too much cost).
Likely cannot book that far not because of the underlying date format, but because of jet-a fuel prices which fluctuate. Airlines typically hedge their near term purchases with longer-term futures
> "What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run."
As someone who writes some very thorough unit tests... and also have had to have mandatory training... I find "this isn't a drill you can run" to be _very_ wrong.
Southwest is a "discount" airline. They do many things to economize, i.e. no assigned seats, they only fly 737s so they don't need to certify pilots or mechanics on any other types, you can only book with them and not with Expedia etc.
It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."
They're also the "friendly airline", they easily have the most personable and friendly staff. I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing, maybe connected to Southwest not having a first-class section.
Usually I fly with Southwest whenever possible without thinking twice about it, but this outage and the outage last year are forcing me to reconsider. Better to deal with rude people than to have my flight delayed..
Yep, the other airlines are in the business of selling "class" and "status," and it's part of their product differentiation strategy to treat you according to how much you pay.
>I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing
Definitely. Normal American customer service is to treat you like human trash, so obviously Southwest has decided to do this differently and so probably does things like only hiring friendly people, training them to be friendly and positive even in difficult situations, and checking on them somehow to make sure they're doing this and not just faking it for the interview and training. Other airlines obviously don't, but it's not just airlines, it's everywhere in American customer-facing business these days. Rudeness is just a normal part of America culture now.
> Rudeness is just a normal part of America culture now.
I have a backpack that has a phone holder on the shoulder strap. When you click your phone in it, the camera is visible. I've had many a situation where someone was about to go full New York on me, but noticed the camera staring at them and toned it down. I've never once been recording.
It's an Orben with a cheap iphone belt holder clipped into the cargo loop on the strap. The belt holder came with the clear protective case I bought at Wal-Mart. I hung the belt hodlder on the strap and it's been great for recording short hikes and downtown excursions. But also, seems to have the effect of making people more polite at customer service counters.
training them to be friendly and positive even in difficult situations
Without having any insight into how Southwest runs things I'd venture to guess that their gate agents and cabin crew have a lot more discretion than their peers at, for example, United. That kind of leeway makes the job much, much easier. United essentially handcuffed their staff leaving few options for dealing with an overbooked flight but to escalate to police involvement.
That's what happened with Dr. Dao in any case. Not long after that shit show, I booked a transcon on American. Turns out it was overbooked, and they were desperate to get people off the plane. The gate agents basically asked how much money it would take to get people to volunteer to take another flight. They got their volunteers, everyone went home happy because American empowered their staff to resolve problems.
A lot of people lay responsibility for Southwest's reputation for customer service at Herb's feet. Personally I found their front line staff to be casual to the point that it bordered on unprofessional (which is something coming from someone allergic to formality), but I'd absolutely believe that much of their positive reputation came from the top. Southwest pilots though, yikes.
With the exception of United I'd say most airlines I've dealt with have been pleasant whether or not I'm flying up front or have status. Yes, even RyanAir. We're in something of a golden age since (with the notable exception of Southwest) American airlines have mostly put their mergers behind them and have mostly shed themselves of CEOs who view employees as adversaries. Take a look at the bad old days of the late 80s through mid 2000s. Smisek. Lorenzo. Parker. Bastards and crooks, all of them. It's difficult to stress just how toxic airline leadership was and how that trickled down for a long time.
Southwest also has pretty good size seats, afaik, relative to other airlines' economy seat size. Plus the whole "you're all the same so we treat you all the same" thing is nice. I do wish there was an international equivalent.
Delta has been more than fine with me. Granted, I'm a Diamond class member now, but even on the way up they were reasonable and provided mostly good service.
> It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."
Meh, it doesn't even have to be "never". It just has to be cost multiplied by frequency is less than the cost to prepare.
If they lose $100m every five years due to a system failure, and it would cost $30m/year to plan for those failures, they it's just cheaper to let it happen.
And I don't mean this in a judgmental, Fight Club-car recall speech kind of way. It's just business reality. At some point every business has to decide that the cost of planning for something is higher than the cost of letting it happen.
What's the value of the reputation risk of a major, very high profile failure?
Sometimes businesses end up on the wrong side of that bet. They see only the costs but not the benefits of preparedness (by the time it fails, there will probably be a different CEO in charge) and make a bad call.
Of course, no argument there. Ideally when you make that kind of decision you take reputational risk into account, as well as, like, is this an existential risk?
The airline industry feels like one where each year it's a different carrier who has some catastrophic scheduling failure. Today, everyone says they're never flying Southwest again. But if you fly semi-regularly then it won't take very long before you don't have any airlines left to fly on.
For people who weren't affected, I doubt very many are even going to remember this. Personally, I remember that this kind of thing has happened recently with other carriers but I couldn't even tell you who.
And people who were affected can mostly be bought off if you need to. Some vouchers & hotel reimbursement and it's just the cost of doing business.
Plus, the airline industry has proven over and over that people are willing to put up with a lot when you have the cheapest prices.
It's different from an industry that's built on reputation and trust. Like, a password manager, the only real thing you're selling is your reputation. Losing trust is a real existential threat. Security costs need to be in the bucket of either "yes, we will do it" or "it's so expensive that if we do it then we don't have a business anyway, so we'll skip it and pray."
Scheduling won't ruin an airlines reputation. Crashing the planes is what ruins an airline. Southwest has only ever had two passenger deaths and one of those was an attempted hijacker beaten to death by other passengers.
FAA has an important role, but market forces won't tolerate an airline that experiences crashes, even rare ones. Airlines are highly incented to be safe, both by regulation and by the market itself.
How badly did SWA's past high profile failures affect them? I'd say not so very much. Yes, some short term damage, but we're talking about an industry where no one stands out from a quality perspective. Everyone is seen as some degree of bad, much like big ISP's. People are used to rotating between airlines and, while those affected this round may shift away, others will get frustrated with their preferred carrier and rotate to Southwest.
I think reputation impacts in this industry from anything other than crashes don't hold much staying power.
All airlines economize. An airline that doesn't is a bankrupt airline because typical industry margins on flight are razor thin.
Southwest isn't a particularly budget airline compared to modern budget carriers like spirit and ryanair that haven't copied the open boarding policy. I suspect the opportunity to upsell seats / luggage and have distinct classes outweighs the turnaround time costs of assigned seating.
Just as a note: they are about to issue a $458M dividend.
They plan to spend $4-4.5B in 2023 on planes.
How much are they spending on system modernization?
It’s funny that you use unit tests as an example of it being possible to run drills for this kind of thing. Unit tests are by their definition not the kind of thing that simulates this kind of failure. Perhaps you have a false sense of security about what you’ve really been testing?
> Unit tests are by their definition not the kind of thing that simulates this kind of failure.
Thorough unit and integration tests include failure modes. Mine include things like "what happens if the OS reports that the storage was unmounted during read/write" (because that was a failure often seen in production with some flaky SAN devices) and "what happens if the server stops responding" (because networks are generally unreliable) and "what happens if invalid (random) data is given" because data corruption is a frequent occurrence for similar reasons.
> Perhaps you have a false sense of security about what you’ve really been testing?
Perhaps. On the other hand I've seen a _lot_ of other developers test only the happy path and call it a day then spend days/weeks/months debugging failures.
Then you are senior enough to know what “ then-CEO now-chairman Gary Kelly” really meant was “I haven’t funded our technology team well enough to have resources to test a scenario like this”.
Or "we decided that the cost to plan for this is so high that it's not even worth testing. If it happens then we're fucked anyway and we'll just eat it."
You can drill the initial failure, but not really the cascading events. In something as large as a global airline you are dependant on 1000s of third parties actions and the weather. No simulated drill is going to be sufficient or realistic. The only way to really mitigate or plan for something like this is multiple layers of segregation so that events in one area have less or no impact on others. Then you could drill total failure in various segments.
Testing reveals the presence of bugs, never their absence. With hindsight you can always feel smugly superior in saying “you should have tested for this”, but there’s an infinitude of things you might need to test, and if you haven’t encountered a failure you didn’t test for, you’re probably just lucky.
Google runs Disaster Recovery Training annually (DiRT) where security teams are tasked with simulating these “black swan” events. Seems like this practice needs to expand to more industries.
At Facebook we would simulate an entire datacenter disappearing.
When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter.
When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.
That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room??
You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
Do you really think the MBA wizards can't figure out some basic pay issues?
It seems you're discounting just how complex HR can be, especially in the face of exigent circumstances. No amount of bonuses will immediately staff up an entire terminal in the face of a massive snowstorm.
> No amount of bonuses will immediately staff up an entire terminal
That is right and thus I would engage line management to figure out business continuity.
> in the face of a massive snowstorm.
It is winter. The storm was tough but not exceptional for the season of winter. Denver did not report tremendous amounts of snow.
Few if any MBAs can get on the ramp and look a line employee in the eye and lend a hand. The wfh keyboard warriors don't know blue collar and therefore are unable to figure this out. MBAs can figure out ways to game their pay. HR can recommend team building 'fun' and non-revenue standby seats, which have minimal value to those employees flying with school age children. Shareholders should demand senior management unemployment applications.
> The storm was tough but not exceptional for the season of winter.
Sure seems like this was an exceptional storm. Widespread, deep cold reaching into Mexico, snow falling across much of the U.S., Buffalo hit with the most snow in 20 years, records set multiple locations.
A couple decades of weather variation is not exceptional, though some will disagree with me when their bonus is yearly. I recall it was cold in Texas last winter. Weather is seasonal and varies.
I wonder if board of directors will compare performance with other airlines.
While they may be able to figure it out, optimizing pay to quality of life at work ratios to ensure long term employee retention and loyalty has certainly not been a priority.
Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
You can certainly handle it better than SWA is handling it. Maybe if they’d actually run a simulation where flights out of one or more cities had to be wholesale canceled, they would be. You can’t fix the situation for everyone, but you can avoid fucking up this badly.
It doesn't seem that farfetched for an airline to run a drill where a given airport is assumed inoperable to see how the system reacts. The expectation shouldn't be the same as the data center failure but you can learn what you aren't doing well enough.
Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.
This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money.
If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.
This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.
That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again."
My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.
Regarding the employees, keep in mind that neither SWA (nor any other airline for that matter) have big software engineering departments. It's all outsourced to either generalist bodyshops for custom/peripheral systems (IBM, Accenture) or specialist shops for core (Amadeus, SABRE)
20 years ago I used to work for an airline. Back then Sabre was nothing more than the mainframe in Tulsa. All the APIs did nothing more than perform automated green screen commands. Has anything changed with Amadeus or Sabre? Or is it still mainframes behind the curtain?
There are two parts to the IT: the airline backend and the distribution backend. For the airline backends, Sabre used to build custom single-tenant ones per airline, on mainframes. The SWA one was very old, hence their inability to charge for bags. The Sabre distribution system is common of course, again on mainframes
Amadeus are eating them up, because their airline backend system is a shared multi-tenant setup, built on commodity hardware. Their distribution system used to be mainframes, but they managed to migrate away in the 2010s.
Sabre is still alive, but only in North America, and Amadeus is slowly chipping away (WN, AC..)
Training for these scenarios may help with responding to true black swan events, like Rick Rescorla's WTC evacuation drills ahead of 9/11. But, nitpicking, if you've predicted something will happen, by way of simulating it, it's not a black swan when it does.
There are cost considerations. Business continuity costs money. Finance firms have significant capital and income to have empty but built out building around around airports for business continuity. Which doesn’t even make sense since they can work from home as proven with covid. Airlines can’t work from home.
Personally one of the basic tenets of my adulthood is realizing how many companies are a hair away from a similar scenario (differing in magnitude from an airline ofc).
Google is made of money, and the reason they are is not because of DiRT. Other industries can't afford the same things that Google can while continuing to be a business.
After canceling 5,400 flights, I can't see how Southwest can afford not to test. Even if they only made $1000 off each flight, that's still $5 million they just lost.
That... and Chaos Engineering in general, and also just generally a forward looking team that identifies threats, vulnerabilities, and risks in the future and works backwards to identify potential mitigating controls.
What's most annoying is that there's plenty of employees on the front line who not only care about testing for this sort of thing, but it actually interests them, they're motivated by it, and they understand the dire reality of what happens – to them, primarily – if the company isn't prepared to handle it.
And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter"
I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
> Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.
Airlines don't have quarterly gains. They regularly go bankrupt and get picked up again, because the country needs airlines and because they have large union contracts.
My wife and I spent a few hours this morning dealing with the cancellation of our return flight. Southwest has long been preferable for me in many cases, including my most flown route. Between the headache of this outage and the apparently dismal state of their operations there's a strong chance I never fly with them again.
Our family was in San Jose last week when our Southwest flight was cancelled.
It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long.
It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.
Asking the government to step in for additional regulation is rarely helpful. For this type of failure, the free market will determine whether processes and tools improve, or whether the status quo is good enough.
Cloud doesn't solve badly designed processes or poorly written software, which seem to be at play with Southwest. Yes, it can help provide more stable infrastructure and there are some (but by no means all) black swan events that can be mitigated simply by throwing more kit at the problem during a surge but it's no silver bullet.
>"What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run."
The unspoken part you have to hear there is "... within the economic model of the airline business".
Business continuity gets exponentially more expensive as you chase the blackest of swans: the sheer volume of plan development and maintenance, developing exercises, table-top vs. walkthrough vs. simulation, assumptions about how many different uncorrelated failures you're prepared for deal with at once etc.
I've no doubt you could run an airline to be as resilient as (say) USAF Air Mobility Command, but no-one could afford the tickets.
What's ironic here is that groups like USAF are constantly pressured to adopt private industry models to be more "economically efficient" and completely ignoring that resiliency is a requirement baked into the high cost. I understand why both take the approaches they do but it seems everyone holds private industry barely running with no resiliency optimizations above all else, which don't make sense in all contexts. Corner cutting is fine in many contexts, especially when you know the side effects of their failures which may be quite insignificant.
On our database systems, we have some date fields for which the default value should never, ever be used and if it is, there is a big problem. All of those dates are set to the dates of well-known natural disasters that happened in the 1800s or earlier.
The thought was that it needs to be something that isn’t believable to a non-technical user seeing it on their computer screen. It turns out that this is not necessarily useful. I listened to a guy talking about some issues with a record; he says “1871? What’s up with that?” And then just moved on as if “well it came out of the computer, must be right” or something.
I think that databases need to have the concept of NaN for dates and time stamps, except that this should be configurable to something like a poop emoji or something like ⁉🆘. It has to be something where your grandma would look at it and confidently say “your computer is broken”
> I think that databases need to have the concept of NaN for dates and time stamps, except that this should be configurable to something like a poop emoji or something like ⁉
So have another column adjacent to the date that stores an enum with an error code with a reason for why the date is missing. My point is, this is business logic, doesn't have a universally applicable definition, and should be handled on a case-by-case basis rather than trying to hard-code it into the date type in the DB.
Default date types, with some columns being nullable, work just fine for my project. I wouldn't want them to be more complicated and force me to consider additional cases, especially if those cases aren't language compatible.
Particularly on mainframe systems like airline reservation systems tend to run on where the Y2k fix in a lot of cases for Cobol was to simply contextually know that certain fields couldn't have been created before 2000, so '00' BCD is simply year 2000.
> "this isn't a drill you can run". And yet, Netflix has chaos Kong do it with regularity.
Netflix and airlines are so different as to make this comparison laughable. The cost of setup and consequence of problems actually being found (ie Federal Regulations) that are not addressable (it's not like SWA didn't know about some of the eventualities), easily outclasses the need for testing every combination of situations. Kong doesn't run anything that has to do with weather turning jet fuel into sludge or 12x pre-staffing in case of massive computer failures along with assessing the possible legal consequences from each locale. The hubris of pretending that physical services on a national scale, is as deterministic as a complex automated system, is unsurprising from a certain crowd, I guess.
Modernization of equipment, hiring more pilots and other employees, investing in updating the code base - how can that be done? It's far more important to keep the stock price high by whatever means necessary, such as using government bailouts to buy back shares.
Investment capitalism is really a garbage system when it comes to building and maintaining basic infrasctructure like transportation, electricity grids, roads and so on. China has demonstrated that convincingly over the past two decades, hasn't it?
When characterized as something that can't be done instead of something they don't know how to do, you know exactly where they are on the Dunning–Kruger curve.
Astonishing. Based on reports from employees on Reddit (https://www.reddit.com/r/SouthwestAirlines/comments/zw32yt/p...), the actual story here is that Southwest has had a complete operational failure and is simply incapable of determining where their crews are, what flights they are eligible for, and associating them with flights on the schedule.
It sounds like there is a semi-automated system which broke somehow, or more likely failed to support the load of cascading changes that resulted from weather disruptions, and they simply don't have the capacity or flexibility to deal with this other than by cancelling flights for three days until they can sort everything out manually.
Reading between the lines a bit, one possible root cause is that their semi-automated system required crew members to update their status by phone when something goes wrong (perhaps with another human in the loop), and the sheer volume of disruption overloaded their phone system and resulted in the automated system becoming completely decoupled from the actual state of reality without sufficient bandwidth in the phone channel to get it back in sync in a reasonable amount of time.
On behalf of all employees: WE ARE SORRY! I will give it to you straight- this meltdown was caused entirely by Southwest. It was triggered by the storm, but the failure to recover quickly is on Southwest 100%. If you are still hearing “weather” almost a week after the storm, it’s not true. Couple main points: 1. Please be patient with us. We desperately want to do everything we can to get you where you’re going. 2. This shitstorm is because the crew scheduling software went belly up and it almost all has to be unraveled over the phone with crew members calling scheduling. If we had better technology which eliminated the need for phone calls, this would have been fixed by now. 3. If you are able to find alternative transportation to your final destination- DO IT. Another airline, bus, train, Lyft, rental car, ANYTHING. Southwest WILL NOT be able to get you to your destination anytime in the next few days. 4. Like I said, it’s gonna take at least a week to get back to normal operations for Southwest. If anyone has questions, I will try to answer them. I work ground ops at one of SWA’s hubs. EDIT FOR FAQs——
Checked bags are currently a disaster. Plan to not see your checked luggage for at least a month. In the interest of 100% transparency, some bags will be 30+ days lost in the system.
Will my flight for X date go out? Next 3 days- plan on a cancellation. 4-7 days- likely to go as scheduled. 7+ days- should see operational recovery.
Incidents like this go a long way toward teaching corporate America that software shouldn't be some forgotten "IT" vertical on your org chart that you staff with random contractors.
Software is core to your business. If you don't invest in it, you're going to pay the price.
People mistaking software as something that just does something is the problem. Software is stored and executable knowledge, so if some component of that breaks down, you need to be able to pull it from somewhere else. Otherwise, you get situations like this.
They are already pretty resentful at us because we have such high salaries compared to the rest of the population... I bet that nobody is willing to give any penny more to their IT departments
According to that post they also have no idea where bags are: "Checked bags are currently a disaster. Plan to not see your checked luggage for at least a month. In the interest of 100% transparency, some bags will be 30+ days lost in the system."
im kind of doubting that thread, not necessarily fraud but not every employee has the full picture
* this memo[1] from the SWA VP thats circulating dated December 21st calling a staffing emergency saying all hell is about to break loose. I'm pretty sure that guy didn't know of the software crash before it happened
* Every single piece of checked in luggage is going to take 30+ days to be found? Weird none of the extensive press has mentioned people not getting their bag. Also its pretty much an airport thing and not an airline thing. Airports are in charge of deplaning your luggage and bringing it to you
* They're running 40% of flights completely manually with 0 scheduling software? That's extremely impressive
> calling a staffing emergency saying all hell is about to break loose
That memo is specifically about ground operations at DEN, and specifically because of the arctic weather conditions and a high number of sick calls and ramp agents that outright quit. You can earn more flipping burgers in Denver than you can as a SWA gate agent with five years of experience. Failing to pay staff is also a management failure.
The scheduling software crashed due to the number of pilots and flight attendants that were out of position and the number of changes that were made to the schedule. I would imagine that there was an overflow in some situation -- i.e. "the number of changed schedules should never exceed 65535" that worked every year until this one. But this system was already known to be unstable, another Reddit comment said that "there are settings you don't change for fear the entire thing will crash." Which it has before in 2016. Not expecting that history will repeat itself and doing something about it is also a management failure.
> Also its pretty much an airport thing and not an airline thing
Absolutely incorrect. While the airport runs the automated conveyance system that gets the bag from where the rampers drop it to the baggage claim, the people that handle the baggage at every manual step in between are SWA employees. If there aren't enough of them, the bags don't make it on the plane on time.
Notice that the people doing this work for Southwest Airlines are all wearing Southwest uniforms.
Like most complicated failures, this was failure with multiple causes and contributing factors. The core of the problem seems to be that management was rent-seeking without making appropriate structural changes to keep up with system load.
I flew out of Denver after several delays/cancels and not only was the baggage area full of bags, the outside gate area was absolutely flooded with bags COVERED in snow. Which means they have been there for days. I'm talking dozens and dozens of haulers loaded to the brim with bags covered in snow. The baggage area was wall to wall bags.
Additionally I was told (by a crew member) Southwest refused to meet wage demands with ramp / fuel crews so they all quit. Apparently they sent them an email basically saying a recession is coming so be thankful you have a job... Meanwhile all other jobs pay more in the area. Anecdote I know, but we did have to wait 2.5 hours for fuel on my flight.
i believe this story. the scheduling system is too fragile and can’t automatically pick up disruptions to crew movement. they’re still manually scheduling some crews, but I think their main problem is they don’t have a way of knowing which crews are where and therefore can’t schedule flights. and it’s not every single piece that’s going to take 30 days, but it seems pretty likely that some bags will be lost that long.
It's not just that. There's complex rules around how frequently crews can work flights and how much time off they need. Delays can cause ripple effects as an hour delay can make an expected available crew unavailable. Then you have a mismatch of available crews and planes and scheduled flights. It's one big thorny ball of spaghetti that even the best designed computer system will struggle to unwravel.
And to top it off, unions fight tooth and nail to make sure this type of information cannot be tracked, which keeps a lot of systems manual. I understand why the unions don't want the data to be captured, but it also makes it hard to optimize physical systems.
Do you have a source? Why would unions fight to keep data from being logged that would ensure that their members aren't overworked? If you're not logging all the working hours how can you even be sure everyone is being paid correctly?
Sorry, wasn’t entirely clear on what data I was talking about.
Unions fight to keep a lot of data from being automated instead of manual because the automated data can be used to build individual performance metrics. For example, some baggage handling unions fought against bag scanners because then you could tell how much work different union members did.
They’re trying to keep from getting optimized to death like an Amazon warehouse worker.
Unfortunately that same automated data is really useful for dealing with situations like this. Once the manually entered data is out of sync with the real world the errors compound and you have cascading failures.
> the software for scheduling is woefully antiquated by at least 20 years. No app/internet options, all manual entry and it has settings that you DO NOT CHANGE for fear of crashing it.
But hey it's cheaper if we don't change it right? /s
(and I kinda can agree with this, like 10% because they might end contracting with some bigger IT company that doesn't care about shipping something that works)
Replacing an airline scheduling system sounds insanely complex. So many moving pieces, tens of millions of physical entities to track per day: crew, passengers, planes, spare parts, luggage, crew maximum allowed shift schedules, open gates, interacting with domestic+international terminals, seat assignments, and so on. Edge cases on edge cases on edge cases developed over years, probably in some god awful language running on a mainframe somewhere before “best practices” were a thing.
Modernizing such a system would take enormous political + real capital. The new system would undoubtedly have many growing pains as formerly resolved problems were not covered in the new system. It’s no surprise nobody would want to touch such a system.
One of the more tangible outcomes from my work on mobile sync is the United scheduling software. Seems like Southwest was too little too late on IT upgrades. https://www.couchbase.com/customers/united-airlines/
How is Couchbase in general and the mobile sync in particular relevant to United not having this disaster right now?
I've worked with airline scheduling software at almost the scale of United (about half the size, for the largest customer). That was using an Oracle database, which was a bit of a pain and a big expense, but worked fine as long as there was competent admins and competent devs. Would Couchbase be disaster-proof even when run by clowns?
Not saying there aren't better and worse choices for databases in any given situation. Just saying that there are lots of them that work perfectly well in competent hands.
Architecture advantages over the phone based system described upthread are obvious. The reason United chose Couchbase is because it’s designed for offline updates. So if a plane, phone, or airport is disconnected they can still do the data entry work and reconcile asynchronously.
But the Southwest system is not "phone-based" by design, I presume? They are going via phone because the decision support system for ops/crew control are down/unusable?
If they've resorted to doing all their scheduling manually with pen and paper, any data pushed to the edge (air crews, ground staff etc) will need to be transmitted in a similarly manual fashion.
The gate/ground/air crews are not making autonomous decisions about who is flying what, when and where. That is decided by some ops/crew control structure in the airline.
There have been plently of reporting about the bag issue, including a Chicago local reporter posting a video of 100's or maybe even 1000's of bags in chicago most of southwest tags on them
It appears there was a MASSIVE disconnect between baggage and people, bags are ending up all over the place even though the people never left.
I've been flying with an airtag in my luggage in Europe since summer this year.
It's been great. My luggage missed a connection in december, when I arrived at the destination I could instantly see that it still was at the other airport. So while the others were waiting for the bags, I could already file a complaint (at the nearly empty complaints queue). Then three hours later I could see bag movement to the tarmac and knew that my bag cought the next flight.
Also, when in a city, it's a lot of fun to say: bring me back to my bag instead of looking up the hotel address.
For anyone with an iphone & checked bags I can really recommend it.
I'm kind of curious about that. If AirTags rely on nearby iPhones to capture tag IDs as they pass by and relay them to Apple, will a ramp worker with an iPhone in his pocket start seeing shorter battery life?
> I'm pretty sure that guy didn't know of the software crash before it happened
Why not? Some things aren't hard to predict, it's like a Ticketmaster employee saying "our site is probably going to go down when the Taylor Swift tour goes live". If you know your systems are deficient then you can predict they'll fail in a crisis, especially if that failure has happened before (as it has for Southwest).
> [it’s] pretty much an airport thing and not an airline thing. Airports are in charge of deplaning your luggage and bringing it to you
That depends on the airline and airport. It’s extremely common for the airline which operates out of a large part of a terminal to do their own ramp work.
Southwest employs thousands of ramp agents to do their own ground ops and under-wing work at their bases.
It's not really a software "crash", it's more like, they know their process can't handle more than a certain amount of disruption before falling apart. And so if you can see that due to weather, etc. a lot of disruption is likely to happen, you can predict there's going to be a problem, and also predict that you simply don't have the staffing and systems in place to do anything about it.
Yeah, and just to add some context for people who might not be familiar with US healthcare, what the VP is asking for is effectively impossible, so this is an order to work sick. The majority of Americans cannot just "go to the doctor" whenever they feel like it, especially 4 days before Christmas. The best most people could do in this situation is call a doctor's office and be told they can get an appointment in a couple months, or go to the emergency room if it's an emergency. So the choice here if you're sick is to waste emergency healthcare workers' time and probably pay extra for unnecessary emergency care, go to work sick, or get fired.
This is obviously intentional, and in many parts of the US totally legal. I don't know specific Colorado laws but there is no federal law against this.
Urgent care facilities typically have less than a 2 hour wait time in every place in the US I've been, and cost about $0-60 per visit depending on insurance.
Scheduling is not an issue, and there is no need to waste an ED's time on such things. Some urgent care spots specifically have a "doctors note" service.
It doesn't make the policy any better (I've worked such jobs), but I don't think there is a need to exaggerate how the process works in real life.
When I'm sick, but not dying, the last thing I want to do is haul myself out to an urgent care unnecessarily to spend money and time, and be in the presence of other sick people. This is detrimental to getting better as soon as possible. Thankfully my employer trusts me, and I don't have to go running to an urgent care every time I get a cold (which they can do nothing about anyway).
A few years back (peak-Covid) I was working for a company helping with temp staffing. It was shocking to me to see the operational expectations for low-wage labor. Clients would pay absolute minimum wage or maybe a dollar/hr more for all sorts of roles then be dumbfounded that workers were flaky.
An airport subcontractor came to us having issues with getting enough baggage workers. Issue was they paid minimum wage, were located 30 minutes outside the city, and there was no public transport. There was no way for the workers to get to the job! The narrative was "these worker are so untrustworthy/lazy/full of excuses." So they would pay a 50% to an agency hoping gig-contractors could save them on single-day contracts rather than just putting that money to better wages or transit perks.
Contempt really does describe the management attitudes I ran into.
Air traffic controllers had a union until 1981, when Reagan's aw-shucks niceness led him to abruptly and unilaterally disband PATCO and fire all the striking controllers. PATCO had endorsed Reagan in the 1980 presidential election.
> "Telemedicine or Telehealth doctor notes will not be accepted"
It's worse than contempt, it's abuse. He is looking for ways to make recovery from illness more difficult than it needs to be, and in turn hope some will come to work when they can't get a "regular" doctor visit in time - during a holiday week.
On the flip side Southwest has stranded what look like tens of thousands of passengers. The memo seems to be focused on curing that problem. It's not clear what the alternative(s) would look like short of ditching all their passengers.
It looks like this is not a fun week for anyone who works at Southwest.
That memo was sent before the storm and before anyone was stranded. It was an attempt to prevent a future crisis. If your goal is to prevent a future crisis, alternatives to ramping up employee abuse 3 days before a big storm might be:
1. Hiring more people (more than 3 days in advance).
2. Paying people more so they don't quit.
3. Giving people paid sick time and not threatening their jobs when they take it so that they don't come to work sick and get a bunch of their colleagues sick.
4. Selling fewer tickets and running fewer flights if you don't have enough capacity to support the current workload.
There's tons of people without bags, and additionally due to the total operational breakdown there is no one to ask about where your bag is and hear any idea about what's going to happen.
> the actual story here is that Southwest has had a complete operational failure and is simply incapable of determining where their crews are, what flights they are eligible for, and associating them with flights on the schedule.
This is amazing. Southwest was famous for its operational efficiency and quality. Companies eagerly learned from them. I wonder what has changed.
Nothing necessarily changed. Efficiency can become fragility, and quality is rarely measured in a way that includes rare events—partly by definition, and partly because we tend to ignore externalities and unlikely scenarios when we make evaluations.
it's definitely epic. I know this is anecdotal but a friend of mine and pilot for Southwest had to pay for his own hotel room after he captained a flight a couple of days ago. SW apparently thought he was in another state (and it wasn't a neighboring one) even though he just flew the flight that they scheduled him on. Not long (hours) after that he was told to drive to the nearest Southwest physical location and check in. Rumor is that all Southwest pilots were told that a physical presence was required to verify location.
It is likely both. Scheduling is quite easy when it goes normally. Scheduling becomes a more difficult task when more exceptions happen. That reddit post is by is some boots-on-the-ground employee, not someone who understands the algorithmic details of SWAs somewhat unique routing challenges.
> Scheduling is quite easy when it goes normally. Scheduling becomes a more difficult task when more exceptions happen.
I think some slack/spare capacity or redundancy somewhere would make a LOT of problems less likely to happen. Running lean only works out when nothing goes wrong. I know I should not be shocked by the extent of human stupidity, but why the "business" types do not get this is bizarre to me.
The airline business is too competitive for labor slack. People put up with spirit’s bullshit because it’s $5 cheaper. Any airline that incorporates slack will immediately find itself without customers. If they make it to a catastrophe it would work out, but it’s extremely unlikely that they will be on business that long.
This is hearsay, but I believe any cancellations due to weather are not subject to fines and fees from the FAA and the airline is not obligated to provide more generous compensation/accommodation for customers. So if there's any reasonable way to blame a failure on inclement weather airline companies will do so without hesitation.
>cancellations due to weather are not subject to fines and fees from the FAA
I thought this too, but apparently there are no federal regulations other than reimbursement for the direct costs.
"Airlines are not required to provide passengers with money or other compensation for costs that fall outside of the cancelled airline ticket and fees tied directly to the airline ticket (such as baggage fees, seat upgrades, etc.) when flights are cancelled." [1]
"Each airline has its own policies about what it will do for customers on bumped or cancelled flights. There are no federal requirements." [2]
However, there are rules around "bumping" passengers on a flight.
"An airline is required to compensate you after involuntarily bumping you from an oversold flight in certain situations. However, there are many situations where you are not entitled to compensation." [3]
How does that post in any way refute the parents point? I agree with the parent that management will try like hell to blame this on the weather- heck they already are. Just check out their press releases.
Yeah I’m a bit confused how the comment even relates to mine. I took it to mean that my assertion that laying blame on the weather for this mess is bogus and a lie. But that assumption may be wrong.
Granted my usually routine flight in 3 days, from BUR to SJC got canceled. And no way to get thru and the usual handy dandy "full refund!" button has been replaced w just an email form with "we'll get back to you!" I sort of believe it
> Southwest has had a complete operational failure and is simply incapable of determining where their crews are, what flights they are eligible for, and associating them with flights on the schedule.
This has to be really hard as most systems were probably built assuming something that smelled like full staffing. Pilots can only fly X hours, and flight attendants usually can't be forced to take overtime ... there have to be similar constraints on ground crew, maintenance and gate workers, too. So when you are planning for an understaffed airline, it has to be extremely difficult to optimize where the failures will occur.
One thing to note is that airline crew scheduling is np-hard (trivially you can reduce Hamiltonian cycle to it). Ak interesting semi-empiricle fact about np-complete problems. Take a decision problem to determine if there is a schedule with k crews. Now there is some k’ that is the minimum such k. For many Np problem problems the difficulty of the decision problem is related to the ratio of k to k’. (Not all decision problems are like this… those that are are on the complexity class APX. As a quick rule of thumb however this works)
In terms of airline scheduling this means that if you more crews than you strictly need, your algorithm is probably going to successfully schedule them. When the number of crews drops below some critical value… all hell breaks loose.
Thinking about this in complexity terms is useful I think. He issue isn’t just bad software… with a small enough number of crews, all software is bad, unless P = NP
I wonder if the difficulty at this point is algorithmic, or if they even have the data to make "reasonable" decisions, never mind optimizing them.
Like what do they do with the bags that are sitting on the plane for cancelled flights? Store them where they are? Figure out their origin and start queuing them to be sent there? Try to get information from the passenger about where to send it?
And then with planes and crews, do they even have "potential" flights where they would be able to reliably seat a worthwhile number of passengers?
I'm sure they know exactly where every plane and every flight crew is at every moment. My point about algorithmic complexity is that even with perfect knowledge, fixing this is a computationally difficult problem. The idea was to show how the properties of algorithms that perform combinatorial optimization actually matter!
> I'm sure they know exactly where every plane and every flight crew is at every moment.
The story being circulated by Sw employees is that this is the failure. The system needs to be manually updated when flights are cancelled or changed, and in this situation with a cascading failure they have completely failed to keep that data accurate. SW employees have said they have spent 24 hours on hold trying to inform the company where they are.
I don’t think this has anything to do with algorithmic complexity.
The statements from employees indicate that SWA does _not_ know where flight crews are at every moment. When a flight is cancelled the downstream information is not updated accordingly. Crew have to call a phone line and speak to another human to update their information and assignments. With the large scale cancellations due to weather, that system was overloaded and fully collapsed. SWA had no choice but to cancel the majority of their flights while they sort out where the crews are at and how to schedule them next. The debacle is a direct result of lack of investment in modern software systems.
heh i wonder if they've asked everyone to use their phones to share their location with their immediate supervisor. Then someone setup a spreadsheet in sharepoint and the supervisors are going at it.
> Like what do they do with the bags that are sitting on the plane for cancelled flights?
Eventually you send them back to the passenger's home address. I've had lost luggage before that turned up on my front porch a week later, some courier service had dropped it off.
yeah it sounds like recovering from this requires standing up a brand new organization in real time. I hope someone is watching the trouble ticket system, imagine that going down...
Yes, theoretically it is NP, if you approach it naively. In practice it is easier to optimize for sub-problems
In practice, crews have their home base, their corresponding qualifications (though for WN they probably only have one or two types)
And you can bet you can start cancelling flights once you don't have enough crew. Problem is, your crew might have gone over their allowed time and them (and your plane) are in Smallville, OH and there are passengers in Chicago waiting to fly.
"Optimizing for subproblems" itself has issues. For instance with a single hub (a complete hub and spoke model), the problem is algorithmically easy. But a) flight distance is much higher, and b) you run the risk of a single weather disaster taking out your entire network, and c) lots of other problems.
My point was to suggest that its useful to understand what's going on in terms of algorithmic complexity. Southwest, because of decisions that they made I should add, is facing an extremely difficult algorithmic problem, and some of the issues of what's going on can be understood by thinking out those problems.
I.E. your algorithms class actually matters
Also: you can work with sub-optimal solutions. Decent approximations exist for many classes of certain problems. Even running simulated annealing or genetic algorithm will get you close to the optimum. I'm sure such a large airliner has a staff with sufficient knowledge of optimization.
I do simulated annealing for a living. Simulated annealing on graphs works great when a) the diameter of the graph is small (which for airlines corresponds to hubbie hub and spoke models) and b). when you can accept relatively approximate solutions. The issue it seems here is that southwest doesn't have enough pilots. Algorithmically, that means they can't accept approximate solutions!
> Algorithmically, that means they can't accept approximate solutions!
Well, of course they can
Not enough pilots make the general problem infeasible, so you have to make do what you have
In practice this means a) flying the schedule as best as you can b) get people/crew/planes back to hubs c) prioritize based on displaced people/cost/other variables
You have to be careful what you mean by approximate. In this context, it means a schedule that requires more than the optimal number of pilots.
Another way of meaning approximate is canceling routes till you can find a schedule that fits the pilots you have. That’s actually a harder problem, and is not CS means by an approximation algorithm. Obviously that’s what SW has to do, but part of the reason why things are so disastrous is that this problem is quite difficult (in terms of algorithmic difficulty)
While this would be an excellent justification for not being able to schedule everything, it's not much of a justification for not being able to schedule 2/3 of your flights on a day with clear weather...
I don't get to exercise my dusty algorithms knowledge nearly enough to follow this nearly as well as I'd like. Are we talking about the bin packing problem and approximation ratios? So, is the intuition here that when the ratio of bins to objects goes up, the worst-case performance for the algorithm goes down into the toilet?
Predictable, assuming they knew which crews in which cities were close to their limit, which it seems like they don't now that the system is down, which is most of the problem.
Incredible to see that 14 months ago there was an almost duplicate outage. Clearly the operations teams are running too lean and don't have enough slack to handle predictable weather/sickness events. Seems like something you'd plan for. Even at the 98%+ uptime range
There's no slack at an airline, it's basically a real time system. Imagine a bottling plant, if one part of the line malfunctions the rest of the line just keeps throwing bottles at the malfunctioning unit until the stop button is hit. There's no way to build in artificial delay to those kinds of systems.
Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air.
> Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air.
You either pay the predictable, ongoing cost or you pay the massive, unpredictable cost like they are currently. There's no free lunch. That is, unless you're an executive looking to boost short-term profits by eliminating "redundancy" and then peacing out before the whole thing collapses from your short-sighted, greedy ineptitude.
People don't like to hear this now, but airlines chose the right thing to optimize for. Air travel is rarely that important to have these redundancies, and they vote with their wallets every time they buy economy tickets. Given that safe is a baseline, most people value price over reliability.
In my experience, every airline will screw you if your not their frequent flyer. Fly once a year? No one is going to waive a fee or refund you anything. Might as well go for cheapest price.
My wife's experience shows SWA as an exception in that.
Is this really a massive cost, though? What does it cost to release some statements to the media and maybe reimburse some hotel receipts in 6 months? Especially balanced against the fuel savings of grounding half your fleet for 6 days. It's absolutely cheaper to run without enough staff and then just deal with the fallout every couple years. No one will remember this. Even the people effected will pick the cheapest flight next time, not matter the airline.
That does bring up an interesting question - are they actually saving on fuel? I would have assumed, given the relatively planned-out nature of most airline operations, that fuel would be something contracted out in advance, and that they may have already bought the fuel whether they use it or not.
Yes and no, they certainly buy fuel in bulk and several weeks/months out, but they're still not flying for a week, so their yearly fuel cost is going to be less than projected.
Airlines are too competing to have extra labor for situations like this. Everyone will book on the other airline that works 99% of the time and you’ll be out of business before a catastrophe hits. “Just add more slack” is not an option that also allows these businesses to be profitable.
There are things called accumulation systems for lines that serve as a buffer to make the line more resilient and able to function if there is a malfunction somewhere in the line. It took me a bit of time to think of the name but I have seen them on large production lines down to small brewery bottling/canning systems.
I had a partner who worked on airline optimization several years ago, as I recall there were standby aircraft in some places that could be deployed to fix problems like a plane needing unexpected maintenance. Or even needing a single label required to be flightworthy. That Southwest doesn't use hubs likely makes it more difficult to recover from this kind of disruption before breaking.
I think the way to do it is how the EU does it, where slack is built in by law (large amounts of compensation for delayed flights). You pay more for each flight in return for predictability, and the race to the bottom is prevented.
Though I'm not sure it would have helped in this case.
But, if customers really valued this predictability, wouldn't a US airline offer it and take all the business?
It's not important enough to consumers, which is why the EU has to regulate it. And I'd be very interested to see how much more predictable they really are for the extra cost...
While that is definitely unfortunate, it looks like the system is working correctly. Aer Lingus is on the hook for millions of euros thanks to EU law, and then it goes after the IT provider to be made whole.
I guess let's wait to see how it works out in the US? The US government seems to be saying they're going to try to do something about this, so it doesn't seem like SWA is going to pay nothing here.
Actually, it appears that Southwest is already saying they'll reimburse tickets for canceled flights and potentially pay out for necessary hotels and alternate arrangements: https://www.southwest.com/html/air/travel-disruption
edit: though there is an extraordinary circumstances exception below. I'm not sure if this would qualify as that given that it's a combination of an adverse weather event and a technical system collapse.
This is not entirely true. Lots of airlines have had events where they need to ground all takeoffs for a couple hours while they put their systems back together. People don't die when airline operations go down so artificial delays are used occasionally. Lots of air travelers have experienced them.
Yes, there's a dollar value attached to everything and everything is being optimized. Southwest was among the pioneers. See the story of the ten minute turn.
I can't for the life of me understand how they're allowed to say this is due to weather (and thus not provide the same kind of compensation they'd have to if it were "Southwest's fault"). Sure, weather is the precipitating (pun intended, sorry) cause, but given that other airlines are almost all at <10% cancellations, clearly Southwest's total lack of operational robustness and competence are the real issue. The CEO himself emailed Southwest staff and explained that one big problem is their antiquated computer systems - that is unequivocally their fault (and very personally the CEO's fault, since it's very much his job to direct the sort of large scale financial investment to do something like overhaul their system).
I hope that DOT follows through on their earlier statement that they're going to investigate this. The CEO of Southwest has ruined more Christmases than the Grinch. Failures like this should trigger incredibly severe consequences (tickets refunded at 4x cost, $20/day paid to any passenger whose luggage is lost until they get it, lost slots in airports) that make it absolutely irrational to operate in the manner Southwest is.
"Weather" covers a lot of sins. You will find that a lot of what gets called (and allowed to be called by the FAA/DOT) "weather" is, as you point out, not really weather, but operational choices.
When a storm hits a city, yes, the first plane that had to wait to take off from the airport can be said to be delayed by weather. Every other plane waiting in line behind that plane, delayed, or because they weren't able to get to that city from other destinations because they run an overly tight schedule, is due to operational choices by the airline.
Some airlines have hubs in places where snow is handled ok. Some don't. Others have them in places that have frequent thunderstorms. Some airlines operate in Hawaii and never have delays.
JetBlue used to fly A321s across the country and in the winter, strong winds would force them to stop in Kansas to refuel. That's "weather" but also the airline's choices about how to operate.
I don't think you'll ever find the FAA/DOT is going to "root cause" what weather means to airlines to be able to blame them for operational / strategic choices. It would be like the police writing up your car accident report and saying that the reason you had a fender bender was because you chose to live so far away from work.
Thus, choose your airlines and roll your dice accordingly for when you want to get where you want to be.
Another example that bit me, flying from east coast to west coast. Storm in the Midwest we can easily avoid but that requires changing the route. Time to change the route with ATC + slightly longer flight time made the pilot time out, flight is cancelled and it’s the last of the night. AA refused to cover anyone’s accommodations because the problem was “weather”
I generally agree, but on the other hand I think the situation here presents a clear case where you could draw the line - if your airline is canceling at a rate that's, let's say, double the national average, you no longer get to claim weather.
It's difficult to punish the kind of individual cases you're describing, but from a game theory perspective that just means that in situations like today's, you bring down the hammer in incredibly punitive fashion in order to make a single systemwide failure like this so costly that it's a no-brainer to upgrade software and keep slack in the system (particularly at high-traffic, high-importance times like the holidays).
> Failures like this should trigger incredibly severe consequences (tickets refunded at 4x cost, $20/day paid to any passenger whose luggage is lost until they get it, lost slots in airports) that make it absolutely irrational to operate in the manner Southwest is
No need to implement the penalties you described if you’re simply trying to align incentives. Their market cap has fallen by $1b, primarily due to this incident.
Their market cap today isn't what matters - if people forget about this and continue to fly Southwest (which, let's be honest, many will), their share price will recover when they announce earnings.
The reality is they save a lot of money by keeping virtually no slack in their system and not doing things like scaling up customer service during the holidays. If they lose some money in flight cancellations, it's still rational for them to operate like this. Plus the reality is they probably won't actually lose much - most people whose flights were cancelled will still end up taking Southwest because it's too expensive (or not possible) to book last minute tickets on other airlines. That means they just push them into the very end of Dec/early Jan when they have open capacity anyway.
I think that's right. Customers have decided that maybe getting charged a couple full fares once every ten years is better than spending 10% extra on every ticket.
Which makes sense. The airlines have decided this as well.
While the capitalist in me agrees with you that this is one the market can sort out (and believe me, I'm going to pay more to not fly Southwest moving forward, at least for 2023), I do think that this is just a case where people are bad at making that calculation. I suspect most people would pay 10% more for even a relatively small reduction in likelihood of this kind of thing happening at Christmas, given the potentially huge costs if it does (missed time with family, being stuck in an airport for days with children, lost payments for hotels and other activities, etc.). People are generally quite poor at properly factoring in low-probability, high-impact events into their decisions, and I think that's happening here.
Now whether the government should step in to protect people from that bias is entirely another question. I would argue yes, but I can very much see the other side that would say to simply let the market sort it out.
> if people forget about this and continue to fly Southwest (which, let's be honest, many will), their share price will recover when they announce earnings.
Well, if you believe this, go buy their stock (LUV). The market disagrees.
I very well may buy some LUV depending on where they end up over the next few days. Remember the Equifax security breech that should have destroyed the reputation of that company in 2017?
Here's some historical stock prices
9/1/2017 $134.64 - everything's fine
9/15/2017 $87.99 - scandal breaks and hits the news
Yeah, I actually did really well after the Deepwater Horizon buying BP stock at the bottom, since the amount of value it lost in market cap exceeded the largest fine in history plus the cost of all the damage and lost oil by something like an order of magnitude.
That said, while in theory I actually think LUV is a decent trade right now, broader market conditions will keep me from messing with it.
A lot of this really does come down to the fact they don't fly hub-and-spoke.
When planes and crew are delayed somewhere, it cascades down the entire line and all the aircraft and crews are now at the wrong airports.
With the hub system, they can do a "return to base" and try to go from there, but with Southwest not having a "base" they have to esentially halt the entire system and reboot it, which is what is going on here.
I have always really wondered how airline scheduling software works. Despite being pretty good with algorithms, I just have no idea how you'd make a system that's robust to weather and mechanical delays.
The field of science (math) that studies this (and applies it) is called "Operations Research"[0] and it's about optimization & planning. About 30yrs ago they started applying it to airline scheduling, here's a few random papers I found:
One important component is slack. Every airline at every airport should have a certain number of crews and airplanes capable of providing service in place of a delayed flight. Running on maximum efficiency for airplanes and staff means unexpected delays will cause cascading failures. Weather can be forecasted, and additional crews can be routed to replace probable future cancelled flights. Temporary staff and increased hours can be utilized for peak demand seasons. We saw similar problems with manufacturing failures when the supply chain became unreliable because of a lack of slack. This type of slack can be seen as an inefficiency and costs money, so it's unsurprising to see budget airlines struggling.
Another important component is disaster recovery. How quickly can the system recover from missed flights? What is the game plan for dealing with crews/airplanes that are out of place. How will they return to normal operations? Often times having a play book everyone is working from can lead to faster recoveries than dealing with each individual crisis as it happens, often with either too much micromanagement from leadership or too little coordination between departments. The play book generates a conciseness before the system is stressed.
> Every airline at every airport should have a certain number of crews and airplanes capable of providing service in place of a delayed flight.
Airline pax are probably not willing to pay for spare standby aircraft and flight and cabin crews at every airport every airline operates from.
Southwest’s original low-cost carrier business innovation was to run an all-737 fleet and make business-wide efforts to optimize for fast ground-turns, in order to get more flights out of each aircraft.
Certainly not, that would destroy profit and competitiveness. Spare airplanes are mostly in for non-essential maintenance, and spare crews can be called up in an hour or two. That's good enough. Catastrophic outages every few years still cost less than building decent redundancy into all operations.
This is partly because airlines are still externalizing a good portion of the cost onto their customers, who need to rebook at short-term pricing. I'd love to see legislation to address this loophole.
Unpopular opinion but airline tickets are way too cheap for what they are doing. My last trip to Vegas, the Uber ride to the airport was more expensive than the airline ticket. My Uber money went toward 1. The driver’s labor, 2. The car and gas, and 3. Uber’s (mostly engineering) overhead. That’s it. And it was like $150! My airline ticket pays for pilots with decades of training, dozens of trained professionals and support agents, baggage handling, security, airport operations, sometimes meal service and entertainment, not to mention the wizardry of launching me 30kft into the air so I can get to another state in an hour. All that for $99?
For comparison, a Greyhound bus on the same route I was about to take a Southwest flight on was about $160 and took 36 hours with 2 transfers compared to $250 on Southwest with one transfer and 5 hours total travel time.
I don't pay for extra standby aircraft, but additional flight availability is why I pick one airline over another. If you choose to fly Spirit, and your flight is canceled or delayed for any reason, you might not make it to your destination for days. With a major carrier, you'll simply be rebooked on the next flight.
Southwest used to be a budget-friendly airline with decent service. Now they're priced as much or more than the other major carriers with the added friction of having to book search flights only on their site.
> One important component is slack. Every airline at every airport should have a certain number of crews and airplanes capable of providing service in place of a delayed flight.
Airlines have crew on "reserve" at all times near bases to handle this problem. They are being paid to sit around and not actually work unless called in. Pilots love to try to get on the reserve list for obvious reasons.
I don't know how Southwest handles reserve, since they don't have "bases" like other airlines do.
> Every airline at every airport should have a certain number of crews and airplanes capable of providing service in place of a delayed flight.
Good luck finding pilots to be “on-call” to fly anywhere in the world (and most commonly to small US cities) on a moments notice, with a jump seat return flight as their way home (after a night in a small city hotel).
You'll find that almost all airlines keep staff on call in various places and with various reporting times, because already at very small scale, you'll have some crew not making it to work for whatever reason all the time.
Maintaining right sized and right placed operational buffers is an entire sub-category of within airline scheduling software/consultancy.
Those buffers will never cover a major disaster of course. They should let you hit your on-time and cancellation targets at smallest possible cost, though.
I worked on the medical resident scheduling problem for a while, and there is a giant body of work on all kinds of staff scheduling problems going back to the 1960s at least.
The two classes of solutions that I considered where optimization solvers (see Gurobi Optimization for example), and meta-heuristics (see the book Metahueristics: From Design to Implementation). If I remember correctly, the people at Gurobi started at a previous company which was spun out of an airline, but I might be confused. All the algorithms in both classes of solutions are so nuanced that it can take years to begin to grasp how their strengths and weaknesses interact with your particular scheduling challenge, and how the way you formulate the problem interacts with the ability of the algorithm to solve it.
All that said, the real problem for me was a human one: If you produce a viable schedule X, the organization involved will always want to alter the rules to stretch the available resources to cover more, and simultaneously all the schedule staff will want more flexibility and nuance in expressing their preferences. You, as the author of scheduling software, are caught between them. Neither side is ever happy with the result.
I occasionally daydream about revisiting resident scheduling (I don't recommend it, the people who use your software leave every year, are not business oriented, and don't understand the complexity of the task until they've tried it on their own their first and only attempt). If I did, I would focus less on algorithms, and more on incentives to reconcile the tension between the organization, which wants to cover the most shifts with the fewest people at the cost of flexibility and preferences, and the staff, who want more flexibility and more preferences satisfied. I think that is the core problem at a business level.
The easiest way to solve the tension is probably to add additional money into the mix - the hard to fill shifts get paid a bonus, etc. someone would figure out how to game it of course but you already have this somewhat when overnights pay more.
Sort of. They bid based on seniority. So if you've been there forever you get the cushy flight that pays a ton. If you just joined to get the worst shift nobody wants (because it's the last one left).
I don't have experience with airline scheduling but I have experience with software in large financial corporations like banks. The situation with banks (at least the ones I worked with) is that there is a huge amount of software maintained mostly by mediocre to bad teams. These teams fail a lot, the software fails a lot, and yet everything seems to keep going.
It is not about code or algorithm quality, it is about procedural side of things -- how the organisation is "programmed" to respond to failures. I use the word "programmed" in a very broad sense -- for me setting up a paper checklist and being able to rely on people to follow it is the same as programming.
I suspect the main difference between airlines and banks is that banks can afford to throw money on the problem and just get things done regardless of how inefficiently.
Airlines are in the much worse position -- they were able to afford being inefficient and throwing money at the problem in the past but can't do it anymore. They work with old, outdated software that wasn't built with efficiency in mind but now don't have funds to change it and are forced to maintain what they have. This may also be the answer to why sometimes they just don't have capacity to react to problem and let it cascade to bring everything to a halt.
Instead of framing this as saying they "don't have" the funds and were "forced" into inaction, another framing that the board of directors must consider is that current SWA leadership failed in their responsibility to recognize, prioritize, and manage the actual needs of their decades old business.
I know a bit about this with airlines - there's a lot of thinking that goes in to making the system appropriately robust but just as much, if not more investment, in optimizing recovery for minimal damage as well. They have optimizers that figure out the minimal damage from a plane being taken out of schedule, and airport halting flights due to weather, or whatever the issue may be. What's cool is the math that goes into "minimal damage" with regards to passengers, crews, bags, etc.
You can poke around at the website for SlickOR (https://www.slickor.com/) to get an idea of the surface level work that goes into this.
I don't understand this. I just flew from West to East Coast, then back, on another airline. Left on time from origin and transfer airport both ways, and arrived early on all flights. The "bomb cyclone" has been gone days, and SW passengers are still stuck, sleeping on floors, or have their bags hundreds of miles away.
As others have pointed out, they don't fly hub-and-spoke.
So when they get planes stranded somewhere, it screws up the entire system and all the planes and all the crew end up at the wrong airports, and it's extremely difficult to get everything back to where it's supposed to be.
They almost have to literally shut down the airline and start it up again, which is what you are seeing now.
In my 20 or so years of travel, Southwest has never been a good airline. But what more boggles my mind is that they've never been the cheapest, either(ignoring spirit and the like). Usually American or United was.
Was that just where I lived (primarily in the east), or am I missing something?
My experience with SW has always been that they're better than everyone else at their pricepoint, and below their pricepoint the savings aren't worth the hassle. They've gotten worse over the years, but they're still consistently the roomiest flights I've been on, their employees have always been friendly and I never really dealt with any major delays with them.
My experiences with Delta were that they were a 50/50 between a nice experience or a "should have just bought a Jet Blue / Spirit ticket" experience. My experiences with United were that anything that wasn't a major route, you were likely going to fly in a tubo-prop coffin and come out with so many aches and pains any money you saved will be spent on advil and massages. They're also the only airline that's managed to lose my luggage twice.
Everyone's experiences will depend greatly on the routes they're flying and their home airport(s). If you're flying on a regional for the legacy airlines, you will be on a smaller plane (though turbo-props are pretty rare these days). The worst is the CRJ200, aka "The Devil's Chariot."
Since Southwest only flies 737s, you'll get a roomier ride than a regional.
There’s no way Southwest hasn’t been the cheapest. I just cannot see that being possible given that they don’t nickel and dime you for everything.
United is by far the worst airline imaginable. I actively will not fly with them after being stuck on an airplane with them for 19 hours straight due to their fuckup, only to have to spend a night in New Jersey at a terrible hotel because we then missed our connecting flight (obviously). In the end, all they offered was a measly certificate that required you to use it at United (and was the equivalent of a mere fraction of the total flight cost).
I haven't flown in 3 or 4 years, if that matters. Most of my flights were to California or DC, so perhaps that matters too.
I know what you mean about fees. I just booked American for my wife last week, and it's horrid now. No free bags anymore, and you have to pay to even pick your economy seats. It was way more infuriating than the last time I'd booked.
I live on the west cost. When I compare United and Southwest, United is cheaper on face value but more expensive with bags. They hide some of total cost in fees.
Throughout the 1990s and early 2000s, Southwest kept hitting the jackpot with fuel price hedging and futures trading. People used to say that they were a hedge fund with an airline attached. They were far and away the cheapest. Now, not so much, unless you regularly check multiple bags.
My family and I are stuck in Long Beach after our flight was cancelled. We called the airline and the next available flight is Saturday. No hotel compensation, no partnerships with other airlines for rebooking on another flight. What a nightmare.
That page describes Southwest's obligations after "controllable cancellations." When a cancellation is outside their control, Southwest only offers to "seek to arrange a discount" on accommodations. They list "weather" and "FAA-required crew duty limitations" as elements outside their control.
SWA has been telling affected passengers to keep their receipts for reimbursement later, so hopefully once things clear up, they will go beyond the letter of their policy to help make things right with their customers. No guarantees, but also not many options otherwise for those of us affected.
Their brand is going to need it, so whatever that cost is will probably be worth it for them.
From personal experience: I used to preferentially use SWA because they were great. Over the years, it seems to have become that every single flight is significantly delayed. Now I preferentially don't use them.
I love not being nickel and dimed by them, and the companion pass makes them significantly cheaper for family travel, but I absolutely avoid holiday travel with them, especially winter holidays.
My datapoint: Had a flight this Thursday from RIC->ATL cancelled this morning. Interesting that they are cancelling flights this far out, where both cities are relatively unaffected by the storm beyond some cold temps.
Was a blessing in disguise for me as I could find a similar flight on a different airline. Hope anyone else flying Southwest can find a way to their destination
The agent this morning told me that they are using the next three days to move staff and planes where they need to be, and the result will be a lot of cancellations when the routes don't serve that. That's why nothing is getting rebooked until Saturday.
> Other issues that have exacerbated the airline's struggle to accommodate the holiday rush include problems with "connecting flight crews to their schedules," Perry said. That issue has made it difficult for employees to access crew scheduling services and get reassignments.
Wait, what issue? Those words didn't actually say anything.
Is this implying that it was a software failure with the scheduling software?
To me, "problems connecting flight crews to their schedules" does not clearly say it's a software issue, but maybe implies it. But you have other information that it is?
Any info on the details? Like, I'd think this is what scheduling software is for...
I feel like this is a good illustration of the problem with "eh, why does technical debt matter, everything is still working isn't it?" It is, until it's not. And then it's too late.
Isn't this hacker news? Weather happened, now lots of planes and flight crews are not at airports they are supposed to be at for the next flight. Southwest probably can only fly specific routes at specific time due to air traffic congestion and regulatory approvals, so can't always fly on a straight line to correct location. By the time they get to the correct place, planes may need maintenance and crews may need sleep. Solving this in close to optimum time is probably an NP-hard problem and company's existing software and employee training is not up to handling this edge case. So there are probably a lot of people pencilling in their best guess for who should go where and calling crews on the phone to let them know. Might take a while to get back to normal.
And the communication is a nightmare. My mom learned that her flight was cancelled Monday via text message, but there is NOTHING on the website indicating the cancellation. I spent 3 hours on hold to get to a rep who rebooked her for a Thursday flight which was also cancelled today. It says to go to a specific southwest.com page, which just takes you to the standard reservation page which, again, has nothing about the cancellation (looks all green). And unless you go to a "travel hardships" page and enter your reservation number AGAIN, you're funneled into their standard cancellation for flight credit (vs. a refund on the original form of payment). It's a total cluster.
Buttigieg has gotten involved, and based on that, I sense a Congressional hearing is in the works. This debacle has affected way too many people at the worst time of year for it to escape public vivisection.
There was a news story a few months ago about some airlines changing the way their crew software works to stop a third party app employees were using to better track their hours Was Southwest one of those airlines?
So in my attempts to get my family home I’ve discovered that rental car company web sites are atrocious.
The direct company sites (not Expedia for example) do not give you the ability to search for vehicles at multiple locations within a mileage radius. You are forced to enter a single location for pickup. This is also true for drop off.
From what I can tell, rental car companies are not setup to easily service a one way rentals, which is what most people need right now. I’ve managed to reserve a car for pickup in KC and drop off in Phoenix for tomorrow. The quote for the two days, over $1200. This isn’t really a web site issue but it gives some context.
Here’s the baffling bit. I make a reservation this morning at Enterprise in a KC suburb and get back a confirmation number, cool. Five minutes later I get a call back that they don’t have any cars available. I ask him why, because I got the confirmation number. He says the website isn’t accurate at the moment. WTF? So when a car is rented and driven off the lot the database doesn’t update? Or worse, the local dealer data is not synced to corporate in real time?
This is a cascading mess. They don’t even have availability in Joplin, MO who’s is 3 hours from KC.
Cars are rented out with the assumption that prior renters will return the cars on time. If that doesn't happen, you will be told that they don't have a car for you.
The reservation system probably overbooks to some extent because a certain percentage of reserved cars are never picked up.
Many rental outlets are franchise operations. It may say Enterprise or Budget on the sign but it's Joe's Car Rental LLC running the place. They may be slow at updating rentals and returns in the system. It's not all one homogenous company.
Car rental inventory is absurd, and renting a box truck is even worse. I've learned over the years that a "reservation" is a best case scenario. - nothing's real until money changes hands. There's also a lot of weird nuance around renting at an airport and renting at an in-town location. And even when you've turned down all the needless upsells and paid for your car, you have to be careful about further sleight of hand!
Earlier this year, I reserved a mid-size vehicle for a few days' drive for a business trip, and after they charged my card, they told me they're bringing around a compact vehicle. I probably should have let it go, the difference in price wasn't that much (and was a work expense) but old habits die hard. "That's not what I paid for - either you can refund me the difference, or you can bring me a mid-size sedan."
They said if I could wait 15 minutes, they'd bump me up from a Hyundai Accent to a larger Toyota, but it wasn't until I sat in the car that I learned that even as a new car, it was as bare-bones as they make them. No CarPlay/Android Auto, the in-car GPS was disabled, no cruise control - I'm sure it would have had manual windows if that were an option. The Hyundai would have been a better ride, I shouldn't have fussed.
But yeah, U-Haul or Ryder or whatever, it's the stone ages there. Years ago, I drove halfway across the state to pick up a sixteen foot truck I'd reserved, and when I got there - "Yeah we don't have a sixteen foot truck, I don't know why it let you reserve that." A smaller truck wasn't an option for what we were moving. I ended up having to call around and hit some local place up.
Penske is the winner there (edit: in my experience, YMMV). We moved from CA to OR in spring 2021. Got a res through UHaul for a box truck, and got a call 24 hours later basically telling me that they were massively overbooked and wouldn't be able to confirm a truck for me until the morning we were supposed to pack the truck up and leave. There's not a snowball's chance in hell that I'm gonna plan for a move with that level of uncertainty, lol.
Called Penske, because I'd always had a good business relationship with them via my job. I was told, "Oh, yeah, we don't do that shit. If you book a truck that day, it's gonna be here, guaranteed."
Sure enough, Penske had it. We packed up the truck, I got on the road, and about 45 minutes later I got a call from UHaul telling me that I could pick up an available truck, but it was three hours away.
I've probably rented box trucks a dozen times in my life for various personal and professional endeavors, including a stint of regular quarterly rentals for several years, and two cross-country moves. At this point Penske gets my money no matter what; I don't even look elsewhere.
I rented cars in Europe, Carribean, Israel and the US. Some things I've learned:
* In the US, Hertz with corporate discounts and status is usually the cheapest and most convenient option. Walk to the car, pick your car from the President's Circle, drive off. Absolutely the best experience I've had with rentals.
* Without status, it's a coin toss: Avis or Hertz, with Avis generally having worse service across the board.
* In Europe, Avis was the worst. In Israel I think it's a franchise, it was all manual entries, long waiting lines. In Portugal I had to wait in line for 1.5 hours to pick up the car because I wasn't Preferred. Their Preferred booth was empty the entire time. All I had to do was validate my drivers license, and they still refused to accept me at Preferred because they print the preferred customers in the
morning. Almost missed my flight.
* Europcar in Israel - cheaper but I wasted an hour in line to _return_ the car.
* In Aruba (I think), Avis doesn't tell you they run shuttles to the airport and you have to wait 30+ min for the shuttle to come around and pick you up.
* I rented 26' trucks from U-Haul twice. Both times went well, though the first truck was very bouncy and the second truck's engine light was on, but otherwise I got from A to B with no issues.
But this all pales in comparison to this car rental company in Moldova (Eastern Europe) that took $100 to book an SUV a month in advance, and then when I arrived they said "we don't have it". The only car they had was $50/day more expensive and instead of apologizing and giving me that car, despite being more expensive, they told me that my only option was to pay more to get this car. They were fixing the A/C outside while I was trying to explain to them that it was a ridiculous demand. A bunch of crooks. I walked away. Sixt was a better experience there, and cheaper than Avis, though 4rent.md is definitely the best option with great customer service.
> Car rental inventory is absurd, and renting a box truck is even worse. I've learned over the years that a "reservation" is a best case scenario. - nothing's real until money changes hands.
As
Seinfeld said, they're good at taking the reservation, but not good at holding it.
My observation of rental car companies is that they have figured out how to run on absolute skeleton crews.
I rented one earlier this month online. When I showed up they handed me a confirmation and told me to pick a car and drive, key was in the cup holder. They had a manned exit barricade to confirm the car you chose, but that was it. I'm not even sure they cared if I picked a car that I didn't reserve, the exit gate person scanned the car and reservation, so it probably would have just updated the reservation on the spot. I don't think it's even possible to predict with certainty what your inventory is going to be 2-3 days out with this sytem.
Just 2 people on the rental side, although I'm sure they have a cleaning/ turnover crew.
> Here’s the baffling bit. I make a reservation this morning at Enterprise in a KC suburb and get back a confirmation number, cool. Five minutes later I get a call back that they don’t have any cars available. I ask him why, because I got the confirmation number. He says the website isn’t accurate at the moment. WTF? So when a car is rented and driven off the lot the database doesn’t update? Or worse, the local dealer data is not synced to corporate in real time?
Same thing happened to me. My wife even called the night before and talked to someone (not at the airport, though) and confirmed. The guy at the pickup had to tell 6 people at 6am that they were only honoring reservations made at least three days prior, and all of them had confirmed numbers.
Likewise, no cars are available within a two hour drive. It looks like the first few days of problems sucked up any available inventory in the system.
Not that this helps but car rental places are like trailer/truck rentals. A particular location will often just be a franchise and own their own vehicles. They'll also handle vehicles from other locations/corporate but they have their own little fleet. They charge one-way rentals based on the likelihood of that vehicle returning to them in some time frame and/or them getting a replacement vehicle while the one-way is rented.
More generally they schedule based on their return schedules. If they have a car scheduled to return at noon and you set your pickup time to one, they assume they'll have that car and be able to rent it to you. If I don't drop it off at noon they can't assume I'll drop it off at 12:01 so they need to let you know it won't be available (since they can't predict my actions).
This is all compounded by agents on the lot renting out cars under the radar or booking agents doing some customer service override. They are incentivized to rent cars, not rent to specific people. For them it doesn't matter if your car is unavailable, it got rented out which made them money.
I’ve found for one way rentals, going between two larger airport locations is the best way to keep the price reasonable (and probably pay some of the difference for Ubers)
Car rentals are a total consumer disaster and have always been this way. It keeps being like this because generally you end up with a car and that's all people need. Special shout out to Sixt that promised me a volvo xc90 over the phone and gave me a cadillac midsize crossover instead when I showed up. The employee at the manhattan location told me they've never even seen a Sixt XC90 at that location.
Just to give you a possible cheat code. Despite them saying you can't when booking...
I've had great luck booking an enterprise rental car from an airport (this is key), and on the booking saying returning to the same airport.
After receiving the car, calling corporate and requesting a different city drop off. The only requirement being that it was dropped off at another airport.
Paid $75 a day for an SUV, from SF and dropped off in Bishop, CA. Reno was the same cost, as well but Bishop made sense. Did this a week ago. Worst case there's an additional cost but is it going to be $1,000 more... no way.
Doesn’t it blow our minds that the U.S. is supposedly the most advanced country in the world, and yet our transportation systems are designed worse than seems imaginable.
On rental cars, I know it’s a meme, but my god it continually surprises me how it seems the agents have to rewrite the mainframe just to hand me keys for a pre-arranged reservation.
At $1200 it's likely cheaper to buy a used car at your origin and sell it at your destination. Availability is also probably less fucked up than the rental market.
For whatever it’s worth, I’ve used National extensively for years (we have a corporate contract) for business and personal, and have generally had positive experiences. The web site has been clear when inventory didn’t exist, haven’t run into the app-reality inconsistency problems.
I was also in STL when this happened. Had to drive 4 hours away to Evansville, IN in order to get a rental car. Just showed up here and they told me I was lucky to reserve it last night, they’re now turning away walk ups at the counter.
SWA averaged about 4100 flights/day in 2021. 5400 flight cancellations is about half of this load.
irrops and cancellations during holidays has unfortunately been normal especially since COVID but I'm guessing fewer employees and the winter storm would cause this to happen.
Oh boy we’re affected by this too. This is infinitely more frustrating with a toddler. Luckily we found a flight out at an airport 2.5hrs away and have a friend who can drive us.
Every rental car in Nebraska is unavailable for the next few days which is crazy.
After several years of this stuff, this is the shape of things to come. Get used to it.
There aren’t enough pilots in the pipeline and your neighborhood is doing everything it can to shut down your local GA airport where people actually learn to fly.
There should be MAJOR fines and penalties here. The reasons stuff like this happens all the time, is because it's just a business expense to the company, while they strand real people in incredibly difficult and often dangerous and occasionally life threatening situations. But for the airline, it's just good business.
Would 100% support an executive getting a long prison sentence on this as well. I know it will never happen but serious consequences are the only way stuff like this stops happening.
Yup sweep it under the rug. SW outsourced all of their development and support to WITCH. The VPs who get condos in Bahamas in return will not ever mention that this is the result of a failure of crew scheduling software because they signed off on the outsourcing.
years ago i interviewed for, and was given an offer, to work on this very system. glad i didn't take it. I feel bad for the devs trying to figure out what's going on.
It's an absolute nightmare of fixed-field records and XML (Southwest uses Amadeus for scheduling but Sabre is just as bad; and AFAIK operations at all airlines are a chaotic hodge podge). Southwest performs all operations in Central Time because nobody's figured out how to reliably add time zones to all systems all at once. This means they have something like a 34 hour operational day, and every flight must take off and land in the same operational day.
It's a satisfying job for the right personality type!
My god. This is terrible but also riveting. I wish there was a website devoted to collecting stories of how these internal systems really work. Airlines. IRS. credit agencies. Etc.
In some recent work I've been doing for an airline, I've worked with 5 different and incompatible ways to represent a passenger, depending on what part of the system you're working in.
You get what you pay for. I suspect, weather permitting, NetJets is still flying.
I fly very rarely. One of the reasons is that I cannot afford to fly on carriers who charge the correct fare for sufficient quality that actually reliably gets you to your destination.
For destinations on the local rail network, I can afford it.
I received a notification that my flight had been cancelled from North Carolina to New York at 12am for a 12pm flight today. My “software incident” senses started tingling and I immediately redeemed credit and rented a car instead (currently on the road). So happy I did that.
Is this a problem with the weather, with the scheduling software, or with the recent shrink then expand of airline demand? Either way, I don't think I'll book a flight with SW for a long, long time.
Are there any airlines that haven’t had similar problems over the last 5-10 years? Seems like they’ve all had issues. And if they haven’t, it’s just a matter of time.
Airlines seem to run their systems at very close to 100% capacity at nearly all times which is sort of the worst approach for resilience in the face of large disasters.
I would hedge with a refundable ticket to a different airline. If Southwest is still having issues you might be able to get the other ticket comped to some degree and still have a flight, if they aren't having issues you can refund the other fare
They need a new CTO, there is no reason any of this could not have been prevented by a solid engineering design. If we can have autonomous vehicles, rockets that can be reused, social media like tiktok, insta, snap, Twitter then no reason we can’t have mission critical systems to support southwest (or any airline for that matter) solved by technology
The comparisons you chose are terrible. We don't have widespread autonomous vehicles. Rockets are complicated but primarily constrained by knowable physics problems. Social media sites are entirely digital systems that do not need to deal with the complexity of the real world and, where they do (like networking connections), they're built on abstractions to manage that complexity.
Logistics problems, especially the logistics of moving humans, are both complicated and complex. You are talking about systems which must interact with manual, human-in-the-loop processes every step of the way. The feedback loops between the digital systems and physical systems are often loose and very costly to tighten (regulations, union rules, passenger behaviors) and the underlying systems that all participants must interact with were some of the very first widespread digital systems.
Today's problem was probably preventable, but don't underestimate how hard the problems become as soon as a system begins to depend on humans and the real world.
I didn’t say that technology can’t fix this, I’m just saying that you’re talking about a complex, physical, adaptive, human in-the-loop system. It doesn’t get much harder than that. You can automat every single part of it, but there’s still human passengers involved so it only takes one asshole to cause a gate delay. There’s no such thing as a perfect system for this. There is better, but there’s not perfect.
Javascript won't put an airplane and the right combinations of crews in a combination of airports no matter how many lines of code you write.
For actual mission critical systems, like delivering packages, UPS Airlines uses a cool concept called a hot spares where an extra airplane is somewhere in the sky, ready to take on slack if there were issues on another plane
I've made this same comment in other replies but none of those examples present the kind of labor issues an airline has. If your essential workers don't show up / quit and operations grind to a halt then no amount of technology is going to fix that
There can be a lot of improvements made to the current experience. Why couldn’t I rebook my cancelled flight through the app? Why couldn’t I be told which baggage claim to reclaim the bags from my cancelled flight? Why did I have to wait two hours to talk to a southwest agent to get a refund? Why did I have to ask for southwest to reimburse me for the damages?
Why couldn’t all of these things be handled easily through the app?
Maybe it won’t prevent my flight from being cancelled, but when they do cancel my flight I’m not left scrambling to figure a plan B so I don’t miss meeting my family for the holidays.
Or they may understand that these extreme cold events are caused by climate change (just one seat in one plane round trip pollutes as much as my whole in 1 year (400kg CO2))
Not sure why my comment is flagged, or why this would be introductory to a nationalistic flamewar. Saying that the US has a lack of emphasis on passenger rail travel has nothing to do with Nationalism. This thread is about Southwest's breakdown and the fallout caused by it. I think it's fair to say that our reliance on air travel is a relevant topic.
Fwiw I use the term "developed world" in quotes because the term is considered offensive to many other nations around the world that don't have the same resources or global presence as the US or large EU nations.
I can see that as a possible interpretation of my statement, thank you for the context. I see that being a mod on this site isn't easy, so thank you for your work. Reading the full guidelines now I can tell I've broken some of them in the past too.
I believe in this case you may have missed one of the more nuanced guidelines when labeling my comment as flamebait: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."
My original comment has the key phrase "with the rest of" which shows the subject (US) , as a part of the set being referenced ("developed world")
In addition to the advantages of trains over flights mentioned by some of the other commenters, trains also:
- Require less specialized crew training, and don't require as many crew members to be equivalently specialized (though you'd be surprised at the depth of training that's still required to crew an Amtrak).
- Can stop, and stop at offloading/transfer points, more easily than planes. In the event of issues this further increases flexibility of the system.
- Have considerably more built-in redundancy than planes, and some built-in swap-ability of components during a trip by removing or replacing cars. I say "some" in italics because I've recently been ... er, very directly acquainted with how critical a single pantograph failure can be to an entire rail line.
- Might, if implemented via state-operated rail or state-sponsored monopoly, offer the ability to replace, deploy, or reroute entire trains in the event of unexpected capacity/mechanical issues. This advantage is a bit of a wash, though, in that we'd probably have more redundancy in air travel if there were fewer larger carriers (this isn't guaranteed and trades off with other issues, but redundancy/flexibility is an advantage of consolidation).
- Are mechanically simpler than planes, and thus require less overhead before being deployed and have fewer "no-go" inspection conditions that can introduce unexpected unavailability.
I'm guessing the GP means that there will hopefully be an increased interest for alternatives to planes that one can take when there are air travel crises like this.
One of the problems with SWA’s case is that cabin crews can work a longer duty day than flight crews and airplanes can’t leave on passenger service without a full complement of both, so the airplane scheduling problem might be more complex.
if i remember my lore correctly it's not a distributed system. It was originally either a mainframe or like AS400 application. I want to say it was related to Saber somehow... maybe they leased it. When i interviewed there years ago they were porting it to some kind of Java stack or maybe they had already ported it and were building it out further. On a tangent, I have to admit, their interview was the best one i've ever had. Very competent people, well rounded process, it was actually more fun than stressful but super challenging too.
/ this was a lonng time ago, like 10+ years, it could be a very very different animal now
I don't think Southwest was ever on Sabre. Their scheduling used to be in-house, now on Amadeus. Computerized operations started with Braniff's software and have ... ah, grown since then. Name almost any technology and it's probably running somewhere in their stack.
Especially when you outsource it to the lowest bidding WITCH shop you can find. Incredible that the US is still pumping out MBAs that see IT as a cost center. If you are of a sufficient size and depend on software to function, you are a software company and need to allocate your resources accordingly.
Here are some crazy things I have encountered.
Rental cars in our city are sold out. Same for cities within two hours drive. The websites will accept reservations, but when you show up, they tell you they have no cars. Because it's the holidays, busses and trains are booked and any flights on other airlines are crazy expensive ($1500 one way). When you are stuck in a city, you are probably truly stuck there.
Your only hope of dealing with SWA is waiting in line at check-in or a gate. The phones don't work. Online chat doesn't work. The lines are long and slow.
Luggage is hit or miss. If your bag was pulled off a plane, you might find it in baggage claim, but most bags are on a plane or on the tarmac. SWA told us it may be 30 days until we get our luggage. They won't pull bags for people, and the agents that we spoke with acknowledged and felt for people who may have had medicine in them.
The workers are as befuddled as the passengers. They have been very nice and as helpful as they can be, but their phones haven't been working and their computer systems have been slow.
On Twitter, someone posted a video of the announcement at Houston Hobby about no flights until the 31st and people keeping their receipts for hotels, etc. They said the same thing at our airport.
People in the airport are so mad. It's unfortunate because it's not constructive. But tempers are flaring, and frustrated passengers who finally get to talk to an agent end up slowing things down because they spend a lot of time trying to hear something they're not going to hear.
Ultimately, this is an operations failure. Companies talk a lot about accountability, but the typical way you hold people accountable is by replacing them with more capable people. It will be very interesting to see if any executives leave SWA over this. If not, I would say that no one was held accountable.
To close, my family and I are fine. This is but a minor speed bump in life. No one is dying, and we will see how SWA takes care of the extra expenditures. Some people aren't so lucky. They have meds in bags, or finances that don't allow them to spend multiple nights in a hotel and get Uber trips for a few days. Hopefully, SWA takes care of them, too.