I’ve been in similar situations, but never without an immediately obvious solution.
That system WILL FAIL. Even as we speak the time-to-failure is shrinking.
Even if one of the solutions described below actually works, you won’t get 100% recovery.
I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup.
Grit your teeth; cover your ass; and get on with it. The clock IS ticking.
I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good.
Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe, it will crash.
I’ve been in similar situations, but never without an immediately obvious solution.
That system WILL FAIL. Even as we speak the time-to-failure is shrinking.
Even if one of the solutions described below actually works, you won’t get 100% recovery.
I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup.
Grit your teeth; cover your ass; and get on with it. The clock IS ticking.