
PyFileFixity: Helping long term storage of data via redundant ECCs - Tomte
https://pypi.org/project/pyFileFixity/
======
oarsinsync
For comparisons with PAR2, I had similar thoughts / concerns / notes, and note
that the author of this has also factored PAR2 into their thinking:

> Parchive (PAR1, PAR2, MultiPar): well known error correction file generator.
> The big advantage of Parchives is that an ecc block depends on multiple
> files: this allows to completely reconstruct a missing file from scratch
> using files that are still available. Works good for most people, but most
> available Parchive generators are not satisfiable for me because 1- they do
> not allow to generate an ecc for a directory tree recursively (except
> MultiPar, and even if it is allowed in the PAR2 specs), 2- they can be very
> slow to generate (even with multiprocessor extensions, because the galois
> field is over 2^16 instead of 2^8, which is very costly), 3- the spec is not
> very resilient to errors and tampering over the ecc file, as it assumes the
> ecc file won’t be corrupted (I also tested, it’s still a bit resilient, but
> it could be a lot more with some tweaking of the spec), 4- it doesn’t allow
> for partial recovery (recovering blocks that we can and pass the others that
> are unrecoverable): with PAR2, a file can be restored fully or it cannot be
> at all.

~~~
StavrosK
It's too bad that one tool exists that's 99% of the way there, but for
whatever reason someone writes a different tool for the extra 1%, and now you
have a split user base.

~~~
justin66
Do you think this would split the existing PAR user base, given that the pros
don't matter much to those users (recursive directory blah blah... just create
a zip file) and the cons (can't replace a missing file) matter a lot?

It's an eccentric user base to begin with, of course. Most archivists and
ordinary users just create multiple backups, and the file integrity concerns
aside from file sharing and archiving are best addressed at the filesystem
level.

~~~
StavrosK
I'm mostly referring to the choice between "using an unmaintained, less
popular utility" vs "using the industry standards that lacks some useful
features".

It would obviously be nice if PAR were faster, for example.

------
hprotagonist
Sigh, and it’s py2 only and hasn’t had a release in 4 years.

An archival programming effort which itself has fallen behind? I hope not,
these sorts of things are pretty important.

~~~
oarsinsync
Good point, I hadn't spotted that. Title should be updated to reflect that
this is (2015)

------
wolf550e
How about the code written by Backblaze (cloud storage and backup company)?
[https://github.com/Backblaze/JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon)

Is it comparable?

~~~
oarsinsync
Yep, they're the same underlying algorithm. I'd wager that the Backblaze code
is probably better tested, by virtue of the scale that they operate at.

Assuming they still use it.

------
nirmalajag
How this compares with HDFS Erasure coding?

[https://hadoop.apache.org/docs/r3.0.0/hadoop-project-
dist/ha...](https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-
hdfs/HDFSErasureCoding.html)

------
metalliqaz
In the old Usenet bin scene (which is actually still going strong) .par2 files
and tools like QuickPar are used to provide redundancy sets for files and they
work very well. It's a wonderful system. They work in a similar way to RAID or
ZFS parity does, and you can create parity blocks for pretty much any set of
files. Powerful and versatile.

------
ris
If it's for storage on optical disks I'm rather fond of dvdisaster
([https://en.wikipedia.org/wiki/Dvdisaster](https://en.wikipedia.org/wiki/Dvdisaster))
which will fill up spare space on a disk with redundant data, specifically
distributing it across the surface to prevent defects in one area leading to
unrecoverable data.

Seems to have become unmaintained though, which is worrying.

------
justin66
It would be quite mad to use this instead of something battle-tested like
PAR2:

[https://en.wikipedia.org/wiki/Parchive](https://en.wikipedia.org/wiki/Parchive)

~~~
myself248
It'd be far better to modify Parchive to include these use-cases in a
compatible way. Other Par2 users might want these functions and not even know
it.

------
jrockway
I really like the idea behind storing error-correction codes at the
application level. Disks should just store and retrieve a chunk of bits and it
should be pushed up to the application level to decide whether or not those
bits were what we wrote. From there, we can make the right decisions about how
durable we really want a file to be.

Instead, people seem to have invested a lot of effort in going the other
direction; let the application have less control and let many magic layers in
the middle figure out the details. We have had filesystems for decades, which
pretend to map names to a list of durable block numbers. Of course, the blocks
are not so durable (nor is the database), so people lose their data all the
time and act like it's normal.

People have a solution to that problem, though, make the block devices more
durable! We can just add software or hardware RAID on top of a bunch of disks,
and now suddenly what the filesystem thinks is just one block is now 3 blocks
sharded across 3 disks. Some magical curator process then tells me when disks
have failed (as though it can detect that with 100% reliability), and I pop in
a new drive and magically everything has been repaired. Of course, in reality,
the entire disk doesn't just disappear all at once; some blocks just return
data with a flipped bit, or all 0s, or an occasional error. My experience in
running RAID-1 arrays under Linux (for my workstation) is that I notice some
errors in the logs and have to manually figure out which disk contains the
correct block (this is why I always use 3 disks) and then dd the block from
the good device to the degraded device. By the time the automatic curator
shows up to detect and repair this problem for me... all three disks have been
RMA'd. It is awful and I'd be surprised to hear a story where someone says
"one of my disks failed, I was notified in a timely manner, and my array was
repaired when I inserted a new disk". It just doesn't track enough data to
actually work, because it ignores the important of error-correction codes.
(Don't get me started on RAID-5. That has some parity, but not Reed-Solomon
codes, and the parity just makes it unlikely that you could ever manually get
the array out of a bad state.)

But even RAID isn't reliable enough, and people don't use it, so of course the
disk pretends it is more durable than it actually is. Spinning rust disks have
some spare sectors. If their internal heuristics that you cannot tune or read
the source code for think something is wrong, it will store some data saying
"now block 42 is actually stored at block 2389473" and my RAID controller,
filesystem, and application never have to know! Hooray! SSDs are even more
egregious; they amplify writes, they garbage collect, they compute proper R-S
codes over the data, and have a multi-core CPU running god-knows-what code
handling it all.

I think it is all crazy. In my dream world, an SSD would just be a bunch of
flash-chips that are not very durable due to the nature of how TLC NAND flash
works. Then my application can decide "this is an important photograph, so I'm
going to shard it across 8 chips on two different controllers and use error-
correction codes that can correct 100% of bit errors". But, that does not
exist.

It all makes me mad because when I worked at Google I actually had control, at
the application level, over all this. There was the underlying disk API that
you could use directly that just stored a block, with the durability of a
datacenter disk drive. Then on top of that were filesystems and databases
where you had to pick the durability for every operation; you could pick
between straight redundancy (write to 3 disks) or whatever level of Reed-
Solomon correction you wanted (all these are tradeoffs between read speed,
write speed, and actual durability).

Meanwhile in the real world, people build crazy things that still lose their
data. ZFS loses data. Cloud block stores lose data. Consumer databases lose
data. I know photographers and videographers that have lost all their work
because of disk failure... they had multiple disks, but their applications did
not uses the disks efficiently in a way that could detect and correct errors.
So you come back a year later, and opening your file just says "sorry, error!"
and you accept your loss and move on. Someday, I hope this can stop. I never
want to watch a YouTube video like this again:
[https://www.youtube.com/watch?v=gSrnXgAmK8k](https://www.youtube.com/watch?v=gSrnXgAmK8k)
(LTT's Windows-based fileserver with hardware RAID fails randomly and cannot
be fixed; all their work gone)

~~~
votepaunchy
> Then my application can decide "this is an important photograph, so I'm
> going to shard it across 8 chips on two different controllers and use error-
> correction codes that can correct 100% of bit errors". But, that does not
> exist.

It’s still not sufficient to backup to a single drive since the device may be
lost or destroyed. And once you are storing in the cloud you have all the
benefits of distributed redundancy and multi-regional support.

~~~
jrockway
Well, yes, you lose the data if you lose all the hardware that is storing your
data. But that's strictly better than the current case, where RAID controller
bugs, SSD firmware bugs, "bit rot", etc. take out your files for absolutely no
good reason.

The cloud is not made of magic. There are some things that work well, like
immutable object stores that are okay with eventual consistency (S3 as an
example), but these stores do not work well for things like databases, or
logs, or editing video.

