System goes down or degrades in some other way at night and important customers with a different timezone get angry, threatening to leave? (happened with us a few times)
But I would'nt use LLM for it due to hallucinations
What's so important that your customers in other timezones feel like waking you up? If they're ready to walk that fast, you can't trust them not to ditch you for an alternative as soon as they find one.
Well, I guess it depends on the business. I forgot to mention that we're B2B. For example, suppose a large food chain or a major bank has an important exam scheduled for their employees on a specific day. If our platform has a blocking bug, no one can proceed (some may be sitting in the class) because the developers are too busy sleeping. Some of our clients are also airplane pilot certification authorities, which have stricter requirements. When there's an alert, you never know if it affects small clients or large clients too.
You don't have to be a missile defense system to require a stable system where devs respond quickly...
But those are the kinds of scenarios where I imagine the sun still came up if comparable disruptions occurred prior to our current era of constant connectivity. We're too invested in the myth that our special problem can't wait half a day.
I'm not sure if you are trolling or genuine, but obviously it is worth it to wake someone up (someone who is specifically paid for being available to be woken up) once it prevents enough costs by resolving the issue now instead of doing so in half a day.
Very few things are so important or costly, and if you're winding up in that situation frequently enough to rob people of their personal time to manually deal with it, clearly something is majorly wrong with this hypothetical critical thing at an architectural level.
Any outage that costs more than paying an engineer to be on-call is worth it. It's not that complex. If an outage blocks 10000 people from doing their work, it's obviously worth it to wake someone up to try to resolve it half a day sooner. (Someone you've been paying specifically for this purpose!)
> rob people of their personal time
Being paid to be on-call is not your personal time.