I use a soft-offline backup for most things: sources push to an intermediate, backups pull from the intermediate, neither source not backup can touch each other directly.
Automated testing for older snapshots is done by verifying checksums made at backup time, and for the latest by pushing fresh checksums from both ends to the middle for comparison (anything with a timestamp older than last backup that differs in checksum indicates an error on one side or the other, or perhaps the intermediate, that needs investigating, as does any file with a timestamp that differs more than the inter-backup gap, or something that unexpectedly doesn't exist in the backup).
I have a real offline backups for a few key bits of data (my main keepass file, encryption & auth details for the backup hosts & process as they don't want to exist in the main backup (that would create a potential hole in the source/backup separation), etc.).
There are actually two lessons there:
1. Be careful what you open to the public internet, including testing to make sure you aren't accidentally leaving open defaults as they are.
2. Backups. Set them up, test them, make sure someone successfully gaining access to the source box(es) can't from there wipe all the backups.