ZFS native encryption is currently broken for encrypted backups

greenavocado · 2024-10-01T06:52:12.000000Z

Thanks for the heads up. I am using regular ZFS on LUKS already. Seconded restic. I researched many backup systems and also appreciate how nice restic is, especially with its compression support nowadays.

BOOSTERHIDROGEN · 2024-10-01T08:23:27.000000Z

How about borg backup?

aborsy · 2024-10-01T08:28:19.000000Z

They are similar in features. Restic is a static binary, so you can copy it with data and it should always work. Restic has a design document, if there are problems in repository, there is knowledge how to remove and rebuild the index. Very nice CLI, and output.

Both have limitations, like they have locks , so you should check on it so that you don’t miss backups. Any easy monitoring solution?

jiripospisil · 2024-10-01T09:41:28.000000Z

> On https://github.com/openzfs/openzfs-docs/issues/494 people unanimously agree that zfs native encryption is broken especially when sending or receiving raw encrypted zfs snapshots, and they blame the zfs leadership for refusing to admit that zfs native encryption is buggy because admitting that it is buggy is bad for the reputation of zfs.

More details on this can be found in a gist from the same author (keep in mind he's a well known zfs commmiter).

https://gist.github.com/rincebrain/622ee4991732774037ff44c67...

benlivengood · 2024-10-01T07:11:36.000000Z

I ran into a short period of time where occasionally sending a raw native-encryption snapshot would fail, but that was the only error, and deleting the failed snapshot resolved the error. Presumably the issue with quotas. That was fixed for me over a year ago and I have half a dozen volumes being snapshotted and sent hourly to three destinations with no errors.

This is on Debian with default kernels and ZoL versions.

I'd still prefer a bit more stability from native encryption.

amano-kenji · 2024-10-01T07:14:21.000000Z

Have you tried restoring from a backup? Given the numerous issues, I suspect restoring from a backup can still fail.

benlivengood · 2024-10-01T17:13:51.000000Z

Yep, individual file checksums match in the destination filesystems, and they get scrubbed routinely to ensure overall durability. I've also been able to send incremental snapshots to a third system from either of the first two, interleaving where the snapshots arrive from, so I am pretty convinced that for my use case the raw snapshot transfer is sufficient. I was also able to change the wrapping passphrase on the origin system and propagate it to the destinations successfully (I can `loadkey` on the destinations with the new passphrase).

JonChesterfield · 2024-10-01T06:53:36.000000Z

I've always felt a little nervous about encrypting backups. The point of the backup is something has gone wrong and you want to get the data back. You don't really want stray bit errors to cascade across vast swathes of the data. Encrypt in transit sure, but maybe not when laying down the bit pattern on the medium at the other end.

lazide · 2024-10-01T07:01:21.000000Z

It Depends(tm).

Some encryption schemes will cascade bit errors (aka a flipped bit somewhere early on will fail everything and you’ll potentially lose access to the entire archive). Those also are terrible for random access. But usually extremely secure.

Most sane ones for backups use a variant of block level encryption where you’ll lose at most a block worth of data from a bit flip.

Which hopefully whatever archiving format you’re using can recover from, since long term storage already has that issue.

Way back, I had issues like this (unstable hardware) and piped tar through encryption (I forget what library - it might have been gpg haha) then rsbep2 to avoid problems. It recovered from corrupted blocks at least twice.

kkfx · 2024-10-01T09:38:28.000000Z

To avoid bitrot the rule is simply having more replicas on different iron. The chance to get ALL broken is obviously low. The big issue is backing up keys or people who memorize passphrases, because for that there is no pure-tech solution.

lazide · 2024-10-01T10:12:39.000000Z

Reed Solomon encoding helps with both. But yes, is not a 100% solution.

amano-kenji · 2024-10-01T07:02:08.000000Z

Restic can detect and fix bitrots in encrypted backups. Restic stores small chunks of files, so only small chunks are affected by bitrots.

aborsy · 2024-10-01T13:17:16.000000Z

It can detect, but cannot fix. It doesn’t have Reed Solomon coding. Very few have, Kopia is the only one that I know of.

ZFS can fix errors too with Raid or copies=2.

amano-kenji · 2024-10-01T22:01:14.000000Z

https://restic.readthedocs.io/en/stable/077_troubleshooting....

aborsy · 2024-10-02T06:06:55.000000Z

This is just repairing the index file, automating what used to be done manually: salvaging the rest of the data that has not been lost hopefully (not restoring lost data).

amano-kenji · 2024-10-02T06:28:30.000000Z

Does restic not try to fetch the pristine data chunks from the source?

Technically, there are two copies of data. One is on the source. The other is on the destination. Thus, restic can be like raid1.

Restic has checksums for chunks. RAID1 with checksum can repair errors.

aborsy · 2024-10-01T06:52:28.000000Z

Encrypted ZFS raw send and receive in several machines including laptop. No issues since the feature was added.

amano-kenji · 2024-10-01T07:03:55.000000Z

It is said that sending and receiving raw encrypted zfs snapshots can work, but it fails for enough people that people should be warned about it.

aborsy · 2024-10-01T08:03:36.000000Z

I am not sure if this is correct. The consensus seems to be, there are a number of related bugs pertaining to ZFS raw send and receive. There seems to be a set of very special circumstances that trigger it. In fact, it’s so rare, that ZFS developers don’t have enough reports and dsta to reproduce and fix it. Moreover, those bugs have not led to data loss (someone may correct me if there are confirmed data loss reports among them).

Otherwise, software always has bugs that you can find their bulletins. Like I use restic and Borg and there are sometimes integrity errors. I have repositories in both with integrity errors in them.