On Monday at 3:30am PDT, we experienced
a hardware failure in our primary encryption hardware device.
The failure cascaded to the backup slave device as well.
Ironically something not far off from that happened to a contract outfit I was working for. They had a telecoms rack rather than a proper one and they couldn't bolt 9x maxed out DL380G2 servers on it as it started to bend. They decided the best approach was to just pile them up on the rack bed.
This was until the one at the bottom needed a RAM upgrade. Cue 6 people carefully trying to lift up 8x DL380's in one go and sliding various OReilly books underneath. The inevitable happened - a hand gave way resulting in a pile of DL380s spread all over the machine room floor and a broken toe from someone in the hosting facility.
Nothing went down though!
Looks like they were sharing encryption keys by replication or shared storage.
The primary fails because of a bug and so it fails over to the backup which flails-over because of the exact same bug, c.f. https://en.wikipedia.org/wiki/Ariane_5_Flight_501