Backups not working when you need them is very very common.
I've heard, experienced (for a client) and read it so very many times. In fact someone told me that pretty much all backups are useless unless you test them frequently (not just once!).
There are lots of tools that do backups - not so many that check if the backup actually works. This might be a business opportunity for someone.
It's not enough to check if the files wrote successfully - you also have to check that you actually backed up the files you need. And that you didn't miss any.
Something like a full restore with unit tests to make sure you actually have the data you care about.
Best pattern I've seen to guarantee backup availability is for upgrades/deployments to occur on the hot failover first, swap the hot failover to primary, move primary to hot failover status, then upgrade what was previously the primary. This, combined with your general data backup (database dump to tape, etc.) gives you much more resilience should something go badly. However, it doesn't take the place of regularly testing your data backups, and only works if you have a full failover environment.
I've heard, experienced (for a client) and read it so very many times. In fact someone told me that pretty much all backups are useless unless you test them frequently (not just once!).
There are lots of tools that do backups - not so many that check if the backup actually works. This might be a business opportunity for someone.
It's not enough to check if the files wrote successfully - you also have to check that you actually backed up the files you need. And that you didn't miss any.
Something like a full restore with unit tests to make sure you actually have the data you care about.