What the author describes here is hard because it's "simple."
What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢
> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice
I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity.
So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline.
That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often.
He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life.
Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity.
Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.
> Scaling to hundreds of thousands of files is not a problem, scaling beyond that and git will start to get slow.
So that's probably insufficient for me by at least a couple orders of magnitude. I'm able to maintain my sanity because snapshots simply capture device state, the NAS collects all snapshots while maintaining their independence, and (so far) borg has been sufficiently scalable to deduplicate any collection I've thrown at it.
Less hard these days. AI is a game changer for learning new technologies. It's like having a highly paid expert available to answer all your questions about your little USB backup. Makes learning how to use properly a new software trivial. And priceless when troubleshooting.
I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!
This caused all sorts of grief at day job where dedupe was useful for a big cache. Conversely I've run ZFS at home for like 15 years at this point without trouble. But this one is an absolute nightmare.
What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢