
Shielding your files with Reed-Solomon codes (2008) - Tomte
http://users.softlab.ntua.gr/~ttsiod/rsbep.html
======
Taek
> the way storage media devices have been going...

My experience is that drives have been getting substantially better in this
regard. If a drive is failing, it will usually collapse into read only mode
and let you know that it is failing, giving you ample time to recover data.

That said, I do think backups are a good idea, and that Reed-Solomon codes are
a good way to protect your data. I also think that managing bitrot is really
the responsibility of the drive itself and maybe of the filesystem. Zfs and
btrfs both offer bitrot protection.

And even solving bitrot, risks like fires, earthquakes, or war (for less
fortunate countries) mean that data should be backed up in multiple physical
locations anyway.

These are problems I've been attempting to address at
[https://sia.tech](https://sia.tech) for the past two years. It's still in
beta, but a full platform should be ready by March.

~~~
mbq
There are also distributed file systems (MooseFS, Gluster, Ceph) which can
duplicate data across physical locations, protect it with checksums, scrub in
a distributed manner and auto-heal, all of that transparently.

~~~
scrupulusalbion
How easy are those to setup and how reliable are they?

~~~
mbq
About hard as setting up any UNIX daemon on a few computers. Reliability is a
complex topic, depends on what you need -- for an anecdotal evidence I
maintain a microscopic (10-20TB/4-6 desktop hosts) Moose deployment for a
distributed home dir use-case over 5 years without any data loss despite two
full computer losses, one disk malfunction and numerous network and power
outages (it never detected a bit-rot event though). There are many serious
success and horror stories one Google query away.

------
kardos
The idea is excellent (I've lost files to bitrot before), but what is the
advantage here over using "Parchives" [1]? There is widespread cross-platform
software available for "par2" format.

[1]
[https://en.wikipedia.org/wiki/Parchive](https://en.wikipedia.org/wiki/Parchive)

~~~
chongli
Par (at least version 1) uses Reed-Solomon codes. It says right there in the
article you linked.

 _Par1 used Reed-Solomon error correction to create new recovery files._

~~~
kardos
Right, par2 does as well [1]. My question is: what advantage does the
freeze/melt thing have?

It looks to me that freeze/melt produces a larger file that contains the
recovery blocks, while par2 stores the recovery blocks in a second file (or
files).

If the recovery bits are stored at the end of the freeze/melt file then you
can still use your frozen file as long as your program doesn't barf on extra
data at the end, while the par2 approach doesn't have that limitation.

[1]
[http://parchive.sourceforge.net/docs/specifications/parity-v...](http://parchive.sourceforge.net/docs/specifications/parity-
volume-spec/article-spec.html)

~~~
chongli
The freeze/melt tool is meant for solving one particular problem: sector
errors on block devices such as hard drives, floppies, and CDs. Par, on the
other hand, was designed for posting to Usenet binary groups where files are
split into many small chunks and the loss of some chunks was so common as to
be expected.

------
13of40
> Storage media are of course block devices, that work or fail on 512-byte
> sector boundaries

It's been my experience that the entire disk typically stops working at once.
"Bad sectors" are a 90's problem.

------
feld
I used to do backups to optical media with freeze.sh and melt.sh. It worked
well; I never had a problem restoring files.

~~~
creshal
Said optimal media already use RS encoding internally and _theoretically_
there shouldn't be problems with these either. Even if you pile several layers
of RS on top of each other you can still have bad luck.

Better to have several backups. :-)

~~~
jeffdubin
We're unlikely to see another mass-adopted optical media format be released,
but if we did, being able to set the amount of error correction myself would
be at the top of my wish list. The option to fill any empty space on the disc
with EC would make me sleep better at night. Yes, there are utilities which
can do this now, but it's not ubiquitous in a way which guarantees I'll have
access to that utility in 20y+.

