Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Long time SRE/DevOps with 20 year of FinTech experience in banks/hedge funds/crypto and having seen many, many outages (some listed here [0]):

- There is almost always a standard operating procedure (SOP)

- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it

- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc

- Doing things outside the "green zone" deployment windows therefore always carries risk

- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways

- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)

0 - https://x.com/alexpotato/status/1215876962809339904

 help



> banks/hedge funds/crypto

Most of us here at HN work (or have worked) at a place where if you screw up a SOP the company could lose lots of money. So we're careful.

That's different than working at a place where if you screw up, people die. It's a totally different environment (or at least it used to be, can't say what happened over the last ten years). There's a lot more 9's, and a lot more consideration of making changes during off periods just in case. I don't believe this was planned for that reason alone.


If you calculate the imputed "uptime" from FAA ground stops since Jan 1, 2000 (there were 3 in the past ~quarter century), you get 99.976%.

In comparison, Stripe has been at 99.999% for many years now.


As a whole, yes, but the individual Stripe services have gone down plenty of times in the the last few months.

More services, more uptime!

"just push it out there and I'll take the risk": https://www.youtube.com/watch?v=U2qgTrqBS_I

(and SkyNet no less)



Duffy has been very loud about doing upgrades faster so this kind of result is predictable.

Doing a video debrief just reinforces his made-for-tv main character syndrome. It only reinforces his behavior.

yep. most IR peeps have plenty of stories where a single event is just a symptom of multiple failures, unfortunate accidents, and alike. most red teams will also regale chained attacks is where it is today. it is rarely a single piece of lock that breaks, it needs several other things to be in the "wrong" state.

Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )

tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.


- The end user in this case has a better idea of what was going on than we do.

- I don't trust anyone in this administration.

- This administration has been acting like a wrecking ball; moving fast and breaking things.

- This administration has been contracting with incompetent private sector companies in blatant acts of corruption and bribery.

- Decision-makers are all drunk on AI these days.

I don't necessarily trust the specific explanation an end user would give for this outage, since they're probably wrong about the specifics. However, when it comes to Duffy, I certainly trust him less:

> Duffy offered a new detail on Saturday, saying that a system overload occurred when contractor L3Harris Technologies Inc. was replacing a section of the air traffic control system’s old copper wire with new fiber wire.

How does upgrading copper to fiber wind up overloading a system when you're adding more bandwidth? And why was this happening in the middle of the day, on a circuit that clearly critical ATC traffic went over? Apparently, the old copper circuits were so overloaded that the bandwidth limitation was critical to imposing request limits, via dropped packets, to prevent overloading the systems on the other end? And once there was enough bandwidth, the servers got effectively DoS'd? That doesn't make any sense.

I'm definitely going to believe the air traffic controllers when they tell me he has to be hiding some other kind of shitshow behind that supposed explanation.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: