It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers.
"You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
I've had that multiple times. As well as the closely related 'that can't possibly have ever worked' and sure enough it never did. Forensics in old codebases with modern tools is always fun.
> As well as the closely related 'that can't possibly have ever worked' and sure enough it never did.
I had one of those, customer is adamant latest version broke some function, I check related code and it hasn't been touched for 7 years, and as written couldn't possibly work. I try and indeed, doesn't work. Yet customer persisted.
Long story short, an unrelated bug in a different module caused the old, non-functioning code to do something entirely different if you had that other module open as well, and the user had disciverdd this and started relying on this emergent functionality.
I had made a change to that other module in the new release and in the process returned the first module to its non-functioning state.
The reason they interacted was of course some global variables. Good times...
By the way, a corollary I encountered, I think with one of the recent AWS meltdowns, is that a paradoxical consequence of designing for "reliability" is that it guarantees that when something does happen, it's going to be bad, because the reliability engineering has done a good job of masking all the smaller faults.
Which means 1. anything that gets through, almost by definition, is going to be bad enough to escape the safeguards, and 2. when things do get bad enough to escape the safeguards, it will likely expose the avalanche of things that were already in a failure state but were being mitigated
The takeaway, which I'm not really sure how to practically make use of, was that if a system isn't observably failing occasionally in small ways, one day it's going to instead fail in a big way
I don't think that's necessarily something rigorously proven but I do think of it sometimes in the face of some mess
That's a fairly common pattern. As frequency of incidents goes down the severity of the average incident goes up. There has to be some underlying mechanism for this (maybe the one you describe but I'm not so sure that's the whole story).
I certainly don't get that cert. I'm seeing a LetsEncrypt cert for idtech.space with various SANs.
# host code.idtech.space
code.idtech.space is an alias for idtech.space.
idtech.space has address 192.99.32.215
idtech.space has IPv6 address 2607:5300:60:47d7::
https://leehite.org/Chimes.htm is the best source of information I've found on chime design and length. They go into great length about the lower octaves and how you can hear them (or not).
I recently created https://immich.pro to partially address this problem. I've got spare compute and storage that I'd like to turn into MRR. While the privacy angle isn't _fantastic_ maybe some people won't care. Could be better than trusting Google/Apple.
Thank you for providing this. In my firefox this article rendered as only the first paragraph then went straight to the comment section and I was very confused about why this was worthy of HN unless to talk about such a short content section!
https://en.wikipedia.org/wiki/Swiss_cheese_model
reply