
Backing up data like the adult I supposedly am - miked85
https://magnusson.io/post/backups/
======
TimWolla
The author uses a systemd timer to schedule their backups. For backups going
to a remote host I prefer adding a little bit of variance to the execution
time to avoid consistently hitting some hotspot.

From the timer I use to backup my server using Borg to rsync.net:

    
    
        [Timer]
        OnUnitActiveSec=24h
        RandomizedDelaySec=1h
    

This will run the backup script every 24 hours with a random delay of up to 1
hour, so every 24.5 hours on average. This causes the job to nicely rotate
around the day.

~~~
ComputerGuru
A loner straightforward solution - albeit not supported by schedulers like
cron with start times denoted by fixed, absolute values - is to use non-
recurring intervals.

Eg run a task at intervals of 86413 seconds

~~~
klodolph
It's why you see a bunch of cron jobs that start off with a random sleep. For
example, certbot's cron:

    
    
        perl -e 'sleep int(rand(43200))' && certbot -q renew

~~~
seanwilson
Seems a bad idea in terms of the accuracy of your logs e.g. so you might not
notice if the command you want to run is starting to run unusually long some
days because of some error.

~~~
klodolph
Another good reason to switch to systemd, IMO.

------
walterbell
With many people using mobile devices, a plug for PhotoSync enabling seamless
photo backup and sync between iOS, Android, Windows, Mac, Linux, local NAS,
iXpand local flash and cloud services. They have both subscription/rental and
lifetime/ownership licenses, [https://www.photosync-
app.com](https://www.photosync-app.com)

iOS storage management has improved with a user-visible "filesystem" and
storage providers allowing edit-in-place, but there's still virtually no
support for backup or rsync. The native iOS Files app is not a robust client
for NAS storage. So far, the best option has been GoodReader (Russian devs)
which implements robust sync (SMB, SFTP & more) within the app, along with
optional in-app strong encryption that goes beyond iOS data protection.
Unencrypted files are visible to other apps,
[https://www.goodreader.com/](https://www.goodreader.com/)

Samsung's iXpand has built an ecosystem of iOS apps that support their custom
protocol for iXpand flash drives via Lightning. Now that iPad enables access
to local storage via USB-C, we need a similar ability to mount a ZFS drive,
even if Apple won't provide this natively in iOS.

With a low-cost x86 SBC like Odroid H2+, an entry-level NAS can be constructed
with Ubuntu ZFS and dual 3.5" drives.

~~~
rsync
"OS storage management has improved with a user-visible "filesystem" and
storage providers allowing edit-in-place, but there's still virtually no
support for backup or rsync. The native iOS Files app is not a robust client
for NAS storage. So far, the best option has been GoodReader (Russian devs)
which implements robust sync (SMB, SFTP & more) within the app, along with
optional in-app strong encryption that goes beyond iOS data protection."

Thank you - this is very interesting.

Although I am a (casual) iphone user (my iphone has never seen my real name or
my real phone number and has _never_ touched rsync.net) I was not aware of the
user-visible filesystem nor was I aware of "Goodreader".

Does this user-visible filesystem allow me to just copy over my entire music
library (which is files and directories, and no knowledge of apple/itunes/ios)
and then let itunes browse it, locally on the phone ? Or do I still need to do
complicated import tasks ?

~~~
sexpositivepriv
What I found is that many old apps have "password protected folders". Some
video apps, some photos apps, some comic readers. This was to keep certain
photos/videos/comic private. Many of those app expose all those "private"
files via the Files app entirely defeating the purpose of having a password on
them.

~~~
walterbell
In-app encryption can keep files opaque to Siri/Spotlight/Files. By default,
Spotlight/Siri will index in-app content, which may (?) also be sent to
iCloud.

------
graton
Like the author I have also been very happy with Borg backup software (
[https://www.borgbackup.org/](https://www.borgbackup.org/) ).

The compression and de-duplication is very useful. A little bit of a learning
curve to get everything up and running, but not too bad.

~~~
aborsy
I also like Borg back up.

I wonder if I am missing something compared to restic?

~~~
magicalhippo
I found restic did not scale well with a large amount of files when I tried it
a few years ago. Has this changed? Is Borg better?

I used to run Crashplan with near-continuous backup for my important files,
and I'm still missing this.

~~~
fidelramos
I don't know what you call large, but my personal restic backup runs weekly,
~250000 files, 2 TB, finishes in 7 hours saturating a 600 Mbps connection over
S3 protocol. I'm very happy with restic.

~~~
magicalhippo
Well I have about 2-3x as many files, but only ~100GB. The main concern was
finding out which files to take backup of, it would take quite a long time.

Like I said I was hoping to get the 10-15 minute intervals I had with
Crashplan.

------
rubatuga
If you want to find out how to backup your iPhone on Linux, I've also made a
guide! It's actually kind of complicated, but it can be fully automated. I
connect my iPhone to my Linux server in my room to do an incremental backup
every night at 5 AM (it also fast charges at 1.8 amps over USB C). I then
create ZFS snapshots every week, since the iPhone backup is an overwrite type.

[https://www.naut.ca/blog/2020/03/20/self-hosting-series-
part...](https://www.naut.ca/blog/2020/03/20/self-hosting-series-
part-4-backup/#compilingidevicebackup2)

~~~
walterbell
Thanks for your self-hosting tutorials for iOS services! The next question is
how to extract individual files from a backup, without needed an iDevice for a
full restore. There are several commercial products sold for this purpose, but
I've not yet seen OSS tools to parse iOS backups.

------
jng
I use TineMachine to three separate idéntica disks: one at home, another one
of the office, and and a third one at my parents place 1000 km away. Pretty
anythingproof. I also have three other disks with three identical copies of my
old archived stuff, in the same locations. Also all code repos are online
(svn, git and hg), and I have most non-code stuff on Dropbox too. Restored
entire machine from TimeMachine once when I upgraded the laptop, ideal
experience. I’m not happy that Covid made me have the office disks at home now
too, but otherwise, I feel pretty safe.

~~~
remote_phone
Is there a way to test if TimeMachine backups are uncorrupted? I backup using
TimeMachine as well but as far as I can tell there is no way to verify a
backup. I’m concerned that at some point my backup will get corrupted and I
won’t know why. This happened to my iPhone backup to iTunes, luckily I had a
iCloud backup.

~~~
saagarjha
Restore it?

~~~
remote_phone
Why on earth would I restore an image back into my MacBook if I don’t know if
it’s corrupt?

~~~
fuzxi
Restore it in a VM.

~~~
remote_phone
Why not have functionality to check to see if the backups are readable?
Instead of forcing users to spin up a vm which most can’t, and do testing etc.
seems like a simple ask. Most credible backup solutions do this.

------
alkonaut
The problem isn't setting up backups, the problem is that you have backups
because something can happen to your data which is 1) you are clumsy or 2)
someone is malicious like a hacker or burglar and 3) physical damage like
harddrive corruption, cosmic rays or fire.

Of these, I find 1) is by far the most common, and 2/3 isn't even close. The
problem with any backup scheme created by myself is: if I couldnt' be trusted
to maintain my data without deleting it, a sure as hell can't be trusted to
set up my own backup scheme without screwing it up.

Example: I have have had regular backups to a NAS, which were then uploaded to
an offsite server were I rotated the data in. But I screwed up a raid config
thing on the NAS after a harddrive faliure, and didn't notice I lost a lot of
data, which after a couple of years had also been removed on the offsite
server.

Basically: to be a good backup solution for me it has to be idiot proof. Zero
manual configuration (If there is a config file or command line anywhere, it's
out). I want a gui tool that gets out of my way and has good defaults, and has
such a huge disk area that it can have effectively write-only semantics. I
want retention for deleted files. Currently I use iDrive which is pretty good
and lets me back up parents computers and so on, in the same 2TB.

~~~
dannyw
I solve this problem by buying a Synology (a popular, idiot proof NAS). Other
trusted solution is QNAP.

Yes, it’s cool to set up a FOSS FreeNAS server with ZFS pools. However, one
look at the forum posts of people losing data due to misconfiguration tells me
it should be considered a toy project.

All of my important files live on the NAS as a source of truth. This means I
passively make sure the data is there, every time I access something.

It’s backed up using Hyper Backup to the cloud, in encrypted format. I verify
restoration once a quarter. I also keep a HDD around that I manually backup to
once a quarter.

~~~
alkonaut
> I solve this problem by buying a Synology

My NAS was a Synology too (The one where I lost a lot of my data, despite
Raid1 + offsite sync). No dry-run restoration from the cloud was my mistake.

~~~
dannyw
How did it fail, if I may ask?

~~~
alkonaut
I don’t remember exactly. It was some small error that I turned into a much
larger error when I tried to fix it.

------
toast0
What software do people like for backing up Windows desktops?

I really want something that ends with a full disk image that's easy to
restore to a new device, runs backups on a schedule (and will run a while
after the next boot if the computer is off at the scheduled time), writes the
images to a unix system on the LAN (either directly, or by writing to SMB),
and doesn't cost an arm and a leg.

~~~
theshrike79
I just went with Backblaze Personal. It's pretty much fire and forget.

Doesn't provide a perfect full disk image, but it does store everything I
need. I've done one full restore from them (fried motherboard from a power
surge) and it went as smoothly as I could expect.

~~~
jaden
Last I checked BackBlaze only kept deleted files for a max of 30 days, making
it a non-starter for my needs. I'm not sure if that's still the case.

~~~
4d66ba06
You can pay more for year long retention now

------
yoz-y
Interesting. Personally I went the other way round and stopped doing dedicated
periodic backups altogether. Photos are on my computer, phone and the cloud.
Code is in repos and the few documents I need in Dropbox. I use a local time
machine disk that I plug in from time to time (which I reformat if I ever need
to reinstall the machine.)

My main reason for ditching periodic backups (backblaze) was that even in
situations where restoring a backup would be useful, I found it easier to just
reinstall the OS and pull a few repos. Nice thing is that this forced me to
automate the machine "setup" so I just have one script that installs my cli
tools of choice and links the correct config files.

------
kneckebrot
> half-assed rsync and shell script abomination

I don’t understand the author’s difficulties with a minimalist bash-wrapped
rsync-based backup. You can even hardlink to unchanged files from a previous
backup to save space.

This is how I wrap rsync:
[https://github.com/kaumanns/snapshot](https://github.com/kaumanns/snapshot)

And regarding file permissions: why not simply use an EXT4 backup drive
instead of an FAT32 one? Non-rhetorical question.

My home network Raspberry has an HDD attached which gets fired up every couple
days for a fresh snapshot of $HOME. The only thing I am missing is redundancy.
And possibly encryption.

~~~
linsomniac
Getting an rsync wrapper to be robust takes some work. The wrapper script I
use evolved over things I found while running it across ~200 hosts nightly for
a couple years. It started as one of those hardlink scripts, but evolved into
using zfs snapshots. My goal was to have it be the ultimate in reliability
though, I wanted it to just work as much as possible, but be quiet unless the
backup failed, at which point it should let me know.

15 years later, nightly backups across maybe 300 machines, this is what I
have:

[https://github.com/tummy-dot-com/tummy-
backup/blob/master/sb...](https://github.com/tummy-dot-com/tummy-
backup/blob/master/sbin/zfsharness)

~~~
tomrod
This is very elegant! Thank you for sharing. I look forward to studying it a
bit deeper.

------
vanpythonista
I'm personally using restic[0] to create encrypted/de-duplicated backups. I
use a local external drive and Backblaze B2 to push the snapshots to. There's
no server to maintain.

The best thing about restic in my opinion is the ability to mount[1] the
snapshots using FUSE to my machine without actually explicitly extracting the
backup to a local directory.

[0]
[https://restic.readthedocs.io/en/latest/index.html](https://restic.readthedocs.io/en/latest/index.html)
[1]
[https://restic.readthedocs.io/en/latest/050_restore.html#res...](https://restic.readthedocs.io/en/latest/050_restore.html#restore-
using-mount)

edit: formatting

~~~
baobabKoodaa
Restic has some failure cases where it claims that backup was successful, when
in fact some of your data was not backed up. This is just about the most
horrifying failure case imaginable.

For example, if you take a backup of a bunch of files which includes TrueCrypt
containers, and then you modify the containers and take a new backup, it will
not back up the new data. Instead, it will look at file metadata to
erroneously conclude that the container has not changed.

Now, some people argue that this is not an issue, because you can use non
default configuration of TrueCrypt and/or Restic to prevent this problem. But
how would a Restic user know that they need to do this?

I don't want to become an expert in the internals of the backup software I'm
using. I just want it to work -- or at least fail in predictable ways.

------
cl3misch
I have been using borg for a while and it has been a joy. But I was under the
impression that you had to trust the server if you aren't mounting the remote
repo, but are running borg on the server too (like the author). Before
commenting that here, I had a quick read of the docs.

Turns out, I was wrong! [1]

> If you use encryption, all data is encrypted on the client before being
> written to the repository. This means that an attacker who manages to
> compromise the host containing an encrypted repository will not be able to
> access any of the data, even while the backup is being made.

borg is even more amazing than I thought.

[1]
[https://borgbackup.readthedocs.io/en/stable/quickstart.html#...](https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-
encryption)

~~~
StavrosK
I used to use Borg but I switched to restic, they're very similar but restic
doesn't require a server (you can use BackBlaze B2).

------
colordrops
I use syncthing to get all my data to a single device, and restic to back
every thing up to backblaze. Restic also encrypts and reduplicates, though I
don't know if it's at the block level.

------
GiantSully
Currently I use node.js + rsync + duplicity to create two backups, and it
works pretty fine. Firstly I create a backup to a local server via node.js and
rsync, then upload them to the remote server via duplicity, which supports
both encryption and compression. Both tasks run periodically and
automatically, and backup files incrementally.

------
grok22
If like me you didn't click all the links on the article and were wondering
where the 100GB for 18$ pricing was (it wasn't on the main pricing page), it
was at:
[https://www.rsync.net/products/attic.html](https://www.rsync.net/products/attic.html)

------
ezconnect
The older I get I got lazier about backups. Now I just use onedrive and
icloud. I know I acn lose them all with a mistake but its alot of convenience
and is cheap.

------
knorker
Does borg prevent attacker verifying the presence of a file? Dedup content
based things may have this problem, and I can't find in the docs that this is
addressed.

File or chunk.

------
m0zg
I wish Backblaze supported Linux. On Mac/Windows it's impossible to beat
Backblaze IMO. It saved my ass against disk failure twice already.

~~~
jimmyjack
I use restic pushing to backblaze from Linux Ubuntu. Works perfect.

~~~
m0zg
You're pushing to B2 though. That'll cost a pretty penny for how much data I
need to store. Come to think of it though maybe I could push to a disk
attached to a Windows machine, which would then back up to Backblaze. Awkward,
but seems like it'd work.

~~~
wjdp
That's very likely the reason they've not bothered with a Linux client yet.
I'd bet the typical Linux user has a hell of a lot more data than the typical
Windows one.

I use B2/restic and only selectively backup things that are either
irreplaceable or I'm willing to pay for. It costs me around 1 USD/month, which
is only ~200GB. I have a lot more data than that on my NAS!

------
nix23
I used borg for a long time but shifted over to Restic (mostly because of
single binary and win/mac/linux/bsd compatibility)

------
RealStickman_
I should maybe have a closer look at borg. Just to learn what alternatives
there are to my current restic + Backblaze B2 setup.

------
beervirus
Just use tarsnap.

~~~
shric
rsync.net: 100 GB for $18/year (no ingress/egress charge)

tarsnap: 100 GB would be $300/year (excluding ingress/egress)

Edit: I believe both support deduplication (borg and tarsnap), but no idea if
one is superior to the other?

