Totally anecdotal but just a month ago I opened a box from my last moving (ended 4 years ago...) and I found a CD-RW labeled "Backup 2001", which survived 5-6 relocations, even an international one.
Well, I put it into an USB DVD player I have aaaand, it starts spinning and spinning and spinning, and nothing shows up in Nautilus. Looking at dmesg I see a few I/O errors on the device.
Then, I try ddrescue [1], start extracting an image, leave it doing its work for a couple of hours and... magic! I have a perfectly mountable disk image!
I got to live again some >20 years old memories, pictures of younger me with my friends. stupid things you would save as a late nerd teen with access to the Internet and a ton of emails in Outlook Express format (guilty!) that I could not open yet. I guess the format is probably not super-hard to parse (most of the content is plain text actually) but since I'm here, if you know of something already existing to convert it to mbox/maildir it would be cool!
Even with ddrescue you might lose a few bits. You will only find out when accessing those specific files and bits.
For my photo history I prefer to keep PAR files next to it. On a running mechanical disc I haven't seen issues when running the repair. I do expect to see some errors in the future and as long as it stays under a certain percentage, things will be fixable.
I started doing this long time in the past, when harddisks were awful. I would often lose mp3 files due to damage. With mp3 you can easily hear it. With jpeg images you can often see damage as well.
If you pay attention to the ddrescue output it is pretty obvious when you are missing a few bytes here and there.
I have long since moved away from optical media, really for anything, but especially archival. Long term storage to me looks like an array of magnetic disks held together with ZFS.
Not much point, way too time consuming. I dumped hundreds of CDs/DVDs as most of them were unreadable. 20% of them might still good, but really no longer had that energy to savage them.
PAR files are a form of Forward Error Correction (FEC) (implemented with Reed Solomon). They can be used to repair some percentage of damaged blocks in a file or set of files. The percentage is determined ahead of time when you generate the FEC (PAR files) and imposes the same amount of storage overhead (i.e., 15% redundancy costs 15% additional disk space with Reed-Solomon).
I recently got rid of all my CD-Rs I'd use for archiving back in the day. Moved everything to a NAS and dumped CDs.
The earliest CDs I had were from 1994 or so, though most were from early 2000s. They were mostly cheapest CD-Rs you could buy back then, so I was expecting most of them be completely unusable.
But most of them still read perfectly. I had maybe half a dozen (out of hundreds) that were completely unreadable, mostly due to peeling, about a dozen that were functionally unreadable, because there were so many unreadable files I couldn't save anything complete out of them, and maybe 10% that had less than 10 unreadable files on them.
I was surprised. I was really expecting almost all the early ones to be dead.
My earliest ones still read. My later ones mostly don't. I'm guessing the difference is the earlier ones were recorded at 2x speed (fastest that existed) while the later ones were recorded much faster (probably 8x or 12x), which means less laser burn in time? Really not sure. Either that or the yamaha drive was just much better than the later model plextor.
I recently sampled a CD-R from late 90s, mostly videos. I couldn't open a few files. Storage: inside home, temperature range from maybe 18 at night to 35 max. I'll give a try to ddrescue.
Same storage conditions here, probably never over 30C though, but I really never cared for its storage either. Give a try to ddrescue, it's pretty straightforward to use, I just ran it with defaults and it worked pretty well, but it has a ton of parameters if you want.
Now I have stored the whole disk image in the cloud (BackBlaze), let see in 2043 if I can still access it!
The problem with cloud storage is that it's a needy thing. You have to pay for it in periodic payments the whole time it exists. And it's also controlled by someone else.
I do really wish there to be a type of storage medium that I could record, store somewhere, and forget about. And then when I need it years later, it would still be perfectly readable. Optical discs are indeed kinda that, but their reliability is hit or miss.
You jest, but here's an idea: print a block of raised letters (like old typesetting) on a 3D resin printer (mirror the characters). Roll a thin slab of potter's clay, say 5"x 8". Press the text against the blank slab, fire the clay in a kiln, and voilà, your bio or other precious document will last like Babylonian clay tablets.
I concur, there is no "fire and forget" solution comparable to what a physical medium gives you. And I could have probably avoided the CD-RW corruption by dumping the whole image on a new CD / DVD / USB storage every now and then.
I also have some external magnetic HDD that is now already ~10 years old, stored in the same conditions as the aforementioned CD, let see how it fares in 2032 :)
Chose CD/DVD as archival medium was one of the big mistakes I've ever made. Most CDs/DVDs failed just after around 10 years. HDDs seem to be much more reliable.
I don't know how common it is to own your living space in the US, but where I'm from, it is fairly common. And I don't think "how expensive per square foot" is applicable when you aren't renting.
But either way, you won't be paying more in rent and/or utilities for having an extra box in your closet.
Unless that one box is the one that causing you to buy/rent more space to store things. In general storage space is very lumpy - people have a fair amount free until they don't and need to get more space - when you get more space you tend to get a lot more. (Of course downsize/decluttering is also an option)
We use square meters in Denmark.
The diameter of a CD/DVD is 120 mm, so you can fit 64 CD's per m² if you order them in a grid (more if you do not)
A CD is 1.2 mm think. The ceiling height in my apartment is ca 2.7m, so I could fit 14400 CD's per m². If a DVD can store 10 GByte, that is ca 144 TByte.
The average rent per square meter is about $20/month (more in Copenhagen, less outside the big cities).
I suspect I can not store 144TByte in the cloud for $20/month.
You might argue that I do have that many DVD's. But then I do have many small corners and shelves that is not much use for anything else. And it would be even more unlikely to be able to store say 14 TByte in the cloud for $2/month.
Google Archive storage is $0.0012/month (3x cheaper than Amazon Glacier, which surprised me). 144TB would be $173/month, so 9x cheaper to keep in your home.
I think your estimate of how many DVDs you can put in a given space is a bit optimistic: you're assuming no protection between discs and that a floor-to-ceiling tower is stable. If instead we imagine putting them in cases that double their thickness and putting collections of them in boxes that add 10% to the total height, though, it's 4x cheaper to keep them in your apartment. But if you use Blu-ray instead of DVD it's 20x cheaper!
I've oddly noticed zero issues reading CD-R's going back to 1999 as recently as last year. I could certainly have some unnoticed bitrot but compressed archives have restored fine.
Depends on the quality of the archival media. Magnetic hard disks aren't without their own failure modes. The magnetic signals on the disk eventually fade causing errors. Motors and arms fail. Cosmic rays flip magnetic state. Or just stray strong magnets if you don't store it right.
Really good archival optical media isn't susceptible to the same cosmic ray it magnetic fade while reducing the chances of disc rot or delamination.
Stray strong magnets haven't been a problem for magnetic hard disks for 30 years. I don't think the fading of the magnetic domains on the disk is a problem over merely geological time scales either, though I'd be interested in finding out if I'm wrong. The mechanical parts are the big deal, and for recent disks (last 25 years) there are also concerns with onboard Flash or other EEPROM losing state.
Cosmic rays will eventually destroy anything that doesn't have an active repair process, including archival optical media, but the time scales are longer than when the Sun is predicted to swallow the Earth.
I've stopped using burned CDs and DVDs like 15 years ago, long before they were considered obsolete, after losing a huge amount of data, including multiple backups of projects. All of them were kept in their own case on a wall CD/DVD holder, no weights, no humidity, no excessive heat or any sun exposition. After only about 5 years like 30% of them were unreadable at all and others developed errors, so as soon as I realized what was going on I backed up everything I could to hard drives in multiple copies, dumped them all and stopped using that media completely.
Printed media last a lot more, but those CD/DVD R/RW to me are too unreliable to be used for anything beyond the occasional OS image install where USB boot isn't available.
Like DNA or living organisms: digital data has to be kept "live" to truly be secure, replicating itself somewhat regularly. Attempts to store it statically are cost-prohibitive or fail at much higher rates than claimed. Worse if the technology is on a down-hill trend because manufacturers focusing on quality are the first to go out of business (higher expenses in a shrinking market).
You can ask the Floppy King about that - near the end of 3.5" floppy manufacturing they saw astronomical rates of failure, 30-50%.
Similarly the quality of CD-Rs definitely took a nose-dive. I'm fairly certain the tail end of "20+ year" CD-R/DVD-R discs won't last as long as claimed but YMMV.
I'm not sure about this being the reality forever. Ever now and then we find some very old scripts or other ancient artifacts containing data from times already forgotten (at least on "basic" human language). If you want something to live eternity just etch the bytes into a stone or print it in some age resistent material.
Of course we can still hope on a easily writable medium that would be also very resistant to ageing.
Given the right context I even think that information produced by humans can outlive earth itself[1]
Sure - but with HDDs, you tend to just move all your stuff during each update cycle, because the new generation then tends to have a multiple of the capacity you've had before.
CDs/DVDs/... feel very much more like a "long-term" storage solution, even when it isn't. Maybe it's because you can put them in shelves and forget about them, much like books?
Plus your backup probably requires dozens of CDs and DVDs, and perhaps they then fit on a handful of 4TB drives, and perhaps now they fit on a single 20TB drive.
For me, my storage needs increase more or less linearly, but storage capacity increases exponentially, making every cycle easier, involving less drives.
This is also true. Having also lost data on two hard drives kept in a drawer for just over a couple years I can totally confirm that, however it was a isolated incident probably also due to some physical issues in the drives, although they weren't old and SMART would report them as 100% healthy. After those two years both drives were completely dead, not just unreadable; ultimately I could bring one to life again after removing and carefully cleaning the controller pcb; however it still required a complete reformat as the data was gone, while the second remained dead. Now I rotate them so that every few years I copy all data to new disks while keeping the old ones until the next rotation: half of my NAS drives become offline backup disks, the other half I use as spare backup disk (with aggressive spindown etc) in some PCs, so I have both live and offline backups. Data should be safe, hopefully:)
Wondering if you do any checking for bitrot? I mean, if some files on one of your disks became corrupt over time how would you know before it becomes too late to recover it from older backups?
I'm no sysadmin so my knowledge in the field is limited. I'm relying on ZFS mirrors which hopefully should be a bit better than the alternatives at bit rot protection, but admittedly if I was hit by corruption at that level that would catch me unprepared. I'm open to suggestions however.
> I'm no sysadmin so my knowledge in the field is limited.
Same here, but my primary backup is also using ZFS mirrors. I have one cronjob that once a month runs 'zpool scrub mypool' to validate all checksums as a bitrot test, and then a day later another one to email me the results of 'zpool status'. If that email contains "state: ONLINE" then no issues were found.
But I've pieced this together myself and would also be open to anyone's suggestions!
If you keep a domestic model running 24x7, you are probably right. But if you just do cold archive, my last check were after around 12 years, all drives spinned up normally without any issue. No noticeable data loss at all.
Out of simple anxiety, I transferred data from a few drives (spanning 2007 to 2015). If there were any issues, neither my file manager nor the SMART status reported any.
My hardrives are in a ZFS z2 array: I have already survived one total failure with no dat loss. I not protected against fire, but overall my data is safer than ever.
Those are BD-R. According to the linked research, those will last 5-10 years, or up to 20 years if you're using a gold metal plate. Either way, they're not close to top performers that last 50 to 100 years and more.
I'd suggest creating multiple backups, including additional mediums such as hard drives and cloud storage (something like S3 Glacier Deep Archive could be relevant), and refresh the media periodically.
They aren't normal BD-Rs nor are they gold plate. They use a different technology for the reflective and recording layers. These discs weren't included in the linked research.
>They literally engrave data into a polycarbonate with high melting point instead of writing to a dye.
Are you sure you're not talking about pressed (ie. mass produced) discs? I find it unlikely that a blu-ray/dvd burner equipped with only a laser can "engrave data into a polycarbonate with high melting point".
> Verbatim M DISC™ optical media is the new standard for digital archival storage. Unlike traditional optical media, which utilize dyes that can break down over time, data stored on an M DISC is engraved on a patented inorganic write layer – it will not fade or deteriorate. This unique engraving process renders these archival grade discs practically impervious to environmental exposure, including light, temperature and humidity.
Me seemed to be a little bit luckier as only 80% seemed problematic after around 10 years. But the result were the same, I had to dump all of them directly except those I did not wish to lose. But even after multiple attempts, still suffered very significant loss.
After reviving a PC i had in the late 90s a couple of weeks ago i investigated this big box of CD-Rs standing besides it (in an unisolated attic in germany... so temperatures from about -5 °C to >35 °C), most of the CDs were burnt in a period from around '96 - 2005, many ... ehm... "off site backups" of games and other software but also personal files.
What can i say... every CD i put in the drive worked flawlessly. I think it sometimes comes down to pure dumb luck.
Quality can vary a lot, back in the late 90's you had certain brands and disc you would buy because of how much extra data you could write to them and also testing to see which disc performed better at higher writing speeds. It was a big deal to some folks back then and you likely bought the high quality ones.
I used to be concerned about optical media longevity. I bought Taiyo-Yuden discs. I checked their integrity. I stored them carefully. But the format became obsolete (for me) long before the media degraded. Recordable discs are just too small for the backup use case I had. Optical drives are still available of course, but they are now rarely standard with a new PC or laptop, and soon will become specialist components, and then it won’t be long before the drives become a greater limit to longevity than the media.
I was a fan of the matte white top printable taiyo udens.
Back in my sharing early days, I probably had like 1000+ dvds of anime, movies, hk film, games etc easily.
I had so much media that I didn't have time to watch it all, let alone watch it again in the future (there are some gems that I actually went out and purchased retail copies).
Plus alot of the shit would get released at higher quality making my archive obsolete (not always)
One day I just chucked it all out.... Years of archiving, downloading, burning ripping.
Exactly the same here, except that I gave all those discs to friends. And I shudder to think how much time I wasted on all that archiving, when it ultimately wasn't worth it at all.
Now, I'm still doing much of the same, but this time downloading movies, TV series, documentaries etc. via Snahp to my archive on an external HDD.
The other day I suddenly had a realisation: Why on earth am I downloading 1080p films to watch sometime in the future when I'm probably going to buy a 4K UHD TV in six months or so from now?
So I need to stop doing so much of this digital hoarding. In the end it's not good for me, and I could instead download whatever I want to watch when I'm actually GOING to watch it.
The problem is that many times, the "originals" (or official sources) become unavailable.
There's a ton of old 80s/90s DOS games and 8-bit games that we still have now because of digital hoarders. If it weren't for them, they'd be lost because the original companies are long gone and most people just trash stuff.
Now look at how much stuff regularly disappears from video services like Netflix and Amazon Prime.
I'm just glad that netflix is still operating their disc rental plan. If you want to watch niche, hard-to-get stuff, that's one of the better options, IMO.
I legally can't recommend anyone copy/rip discs that they rent.
In my case it was early MMA fights, I was obssessed with martial arts in general, and up to 2007 I literally had a 500+ collection of burned CDs and DVDs, with every single fight from every major promotion in the world, catalogued by date, fighter's involved, styles, results etc. Until someday I realized that I probably would never rewatch any of those fights, I would simply watch the new ones that are happening everyday. Hoarding is hoarding, be it of "real things" be it digital ones. Some time later I gave all of my collection to a BJJ colleague of mine, and I never burned another CD since. Life's too short, and accumulating stuff is just a massive waste of time.
I recently left my home for an apartment and came to the same conclusion about accumulated stuff. Watches, guns, calculators, shirts, shoes were all bad but worst of all trying to cram 1000 lbs of books into a two bedroom apartment is ridiculous. I am working on minimizing but it is hard to let go of these objects that I love. Thankfully I have long since moved my data hoard off to cloud storage so at least that only costs money.
In my case I've got the complete collection from The Twilight Zone and Star Trek:TNG among the movies. Everything else it's just a rehash from these scripts.
> Optical drives are still available of course, but they are now rarely standard with a new PC or laptop, and soon will become specialist components, and then it won’t be long before the drives become a greater limit to longevity than the media.
IMO, the #1 use case for optical drives is probably playing music in the car. For anything old enough that it doesn't have an aux port, it's the best solution.
* Tape adapters add a lot of noise. Maybe different drives work better than what I've got, but I tried and it's totally unworkable
* Short range FM adapters are terrible. Similar noise to tape adapters and the quality is really prone to degradation depending on the EM environment.
The stock of older, working cars is quite large. Eventually it'll get used up (outside of enthusiasts), but that'll probably take a few decades longer.
And as long as people are listening to CDs in their cars, there'll be a demand for the ability to burn them.
If you’re in the US, try Crutchfield. Not the best prices if you happen to own an audio shop and already know how to do everything, but they include instructions on how to do everything (including take apart your dash) and phone support for the lifetime of the product, and for little or no extra include all the pieces needed to interface your aftermarket system with the stock harness.
I have a 2001 car that has been using a Bluetooth audio and hands-free setup from them for over ten years. (It’s old enough that “you can control your iPod with it if you plug it it” was a selling point.)
> IMO, the #1 use case for optical drives is probably playing music in the car. For anything old enough that it doesn't have an aux port, it's the best solution.
Not even that old. A friend of mine bought a BMW MY2020 and it still has a CD-player (it was an extra, but it was at least still available in 2020).
I wouldn't be surprised if you could still order new car, today, with a CD player.
But I have to imagine that people who have a choice between AUX/bluetooth and CDs, and choose CDs are in the minority.
It's kind of a pain to install a CD burning app (which has long since atrophied from most music programs), plug in a removable drive, etc. I only do it because the alternatives are much worse.
My 2012 car supports reading mp3s off a CD which greatly improved the utility for me.
I kind of prefer it over the AUX jack since I hate messing with a touchscreen phone while driving. The radio has actual buttons and since it only does one thing it doesn't need to be messed with much to begin with.
Just my photo library is over 1.5 TiB, there just not way to have long-term offline backups of that, you have no choice but to run and maintain a file server and cloud backups
I've had really bad luck with the reliability of mobile HDDs in storage. Try to use them after 3-4 years and they just die. Maybe I've just been unlucky?
You do realize that HDDs can randomly break, right? To keep a proper backup you need to have multiple copies of all your data and also regularly access / copy it.
After quite a bit of research into this about a year back I decided to archive everything on archive-grade CD-Rs. 20-50 years is more than enough and realistically in the correct climate (a safe or something that humidity would have trouble getting into along with dessicants, etc) could last much longer. I evaluated magnetic tape, HDDs, and a few other options are CD-Rs are as good as magnetic tape (in longevity) and will likely have readers well into the next century.
But you have to be very careful. There are a few brands, in particular Taiyo Yuden that are the absolute best. CMC Pro is a related brand with the same technology/longevity (and easier to get). You will pay quadruple the cost per CD but its worth every penny.
Avoid Memorex, JVC, etc. Basically any brand you can pick up at your local big box retailer.
TY closed down their own disc manufacturing in 2015. Nowadays the CMC Pro line (with 'Powered by TY Technology' branding) is the closest you're going to get.
I don't unfortunately. I use CMC Pro now which is highly regarded. They're "powered by TY technology" which I took to mean essentially white labeled TY, made in Taiwan, so I also assumed that CMC Pro was the actual producer of TY and now that TY is defunct CMC Pro took over selling stuff.
I've always only bought Taiyo Yuden media since ~2000. Every CD-R and DVD-R I still have that I've tried is still readable.
Taiyo Yuden at least used to be white labeled sometimes, so you'd just look up online what white-label media at Best Buy or CompUSA at the time was Taiyo Yuden.
I haven't burned a disc in probably close to a decade at this point though. SpiderOak is my backup.
I think I have a lifetime supply of TY media. I hardly ever use it. I don't even know if it is CD or DVD. I'd have to look. I wonder if there is a shelf life for the media before it is recorded.
I'm in the spinning rust camp. I use the ZFS filesystem which protects against bit rot and I usually migrate to larger drives every several years. And I have an off site mirror of my local storage. My only concern is what happens to it when I'm no longer able to manage it. I guess I need to leave instructions.
Honestly never came across it in my research. I used music producers and professional archivists as my leaping off point and for them TY and CMC Pro seemed to be the bread and butter. I'll take a look at these though!
There's allegedly nothing different special about blu-ray M-DISCs
>Digging further around the M-disc site, it is noted that their section on longevity testing actually features no data about the BD-R discs at all. Indeed, if there were tests, the reports were not actually featured on the site at all. Further poking around in M-Disc Technology pages shows only references to their DVD products. In fact, it seems their patented “etched in stone” material may only apply to their DVD products.
I have some M-DISC DVD's from ~15 years ago that still read fine.
That said I still have a lot of cheap CDs from 20 years ago that still read fine as well.
I've had a 25GB M-DISC BD-R kicking around on my desk without a sleeve or case for a while now to test durability. Sometimes I take it outside and leave it on my patio table for a few days to add some extra wear. I slide it around the desk, pick it up with normal greasy/dirty hands and handle it non-gently, flex it a bit, wash it off in the sink, and wipe it off with a cotton cloth. After four years of this abuse so far it still hasn't lost a single bit of data. I have noticed some of it takes a bit longer to get a good read so I'll probably lose data soon but this has been a hell of a lot of abuse so far.
yep, I have about 1 TB of data I backed up 5 years ago that I was looking thru. I used the BDXL 100 GB disks.
I’m a huge fan of mdisk but I’m not sure it makes much sense in the era of cheap cloud storage. There are many companies now that will give TB of storage for 9$/month. Also there is cheap archival storage on AWS.
I just started using M-Discs last month. Bought a compatible burner, a Pioneer Blu-ray Disc (BDR-XD07B), CAD $170. I'm pleased to report that my burning software (ImgBurn) worked fine with the new Pioneer DVD drive.
At $10 a pop, one is very prudent when choosing what to backup.
When I was using dual-layer DVD+R blanks for off-site backups, the folk wisdom on blanks I heard from someone who burnt discs a lot was: to get a particular type and brand, and make sure the unit had been manufactured in Singapore. The country/factory of manufacture was indicated on the packaging, but the SKU and UPC didn't reflect that.
More recently, I looked into re-adding optical disks (this time, M-Disc at BD capacities) as one backup medium for current home servers, but the tech manufacturers just seemed even more user-hostile and Linux-hostile than they used to be. Maybe it's due to the intersection with piracy, maybe it's due to market forces of consumers who aren't very savvy about their own interests, but whatever the reason, it didn't look like an attractive option.
(Low-level burning software for optical drives was always crazy, and required some tricky alchemy by saintly open source programmers, as well as a user tolerance for burning the occasional "coaster" (failed burn). I layered my own, simpler, software atop their critical work. But I'm not interested in reinvesting in a category of media that in some ways seems to be getting worse, or even customer-hostile, at the same time that it's fallen out of popularity. If someone has better information, about how some corner of this is rock-solid, non-hostile, and otherwise viable, then I'd be happy to reconsider.)
I do backups of my home servers to BD-RE discs (BDXL 100GB) once a week from ZFS storage. The only problem is poor drive quality - for past five years I changed two dives (Asus branded, all different models). Seems they are not made for regular writes. So, the weak point is not the media itself but availability of dives which may just be missing in 20 years perspective.
All of the CDs in my old collection are fine, but I’ve had major problems with my DVD collection. I haven’t spent any time trying to figure it out, but it looks like some kind of unusual mold that I’ve never seen or some kind of deterioration of the substrate occurred inside some of the discs and they became completely unplayable. Attempts at cleaning completely failed to solve the problem. Is there really a major difference between DVD and CD manufacturing on a commercial level?
As explained in the article DVDs are made by sandwiching the dye and reflective layers between two polycarbonate layers, which are then glued together. The adhesive used is sometimes prone to failure, causing delamination of the disc (this can happen with pressed DVDs as well, I have a few Xbox 360 games that became unreadable due to moisture getting between the layers). CDs do not suffer from this issue as the data layer is placed directly under the label, with no need for an intermediate polycarbonate layer.
Print media is often considered 'more permanent' that other digital storage media. But consider: every courthouse I ever visited had records 'back to the fire', the time the records room burned. Every one.
Considering the flammability of paper multiplied by the probability of a fire annually gives you the means to come up with a mean-time-between-failure for paper records. And I'd guess it isn't very long - maybe 50 years to a century.
Unless you have ideal storage conditions, print media won't last more than a few hundred years - and that is with acid free paper, acid treated paper won't last that long.
There is a reason ancient documents are found in Egypt "all the time" and almost no place else in the world: Egypt is an ideal climate to store paper. The rest of the world paper will decay in a few hundred years. (of course modern technology can create better conditions - if we choose to keep it running)
Sprinklers, as well as waterless fire suppression systems, are fairly recent inventions so their mettle has not been really tested on the time scale of document archiving.
There was a panic circa 1980 about the longevity of color cinema film. Martin Scorsese pointed out the cultural loss but it was also a big dollar and cents loss because home video was on its way as a way to turn old movies into cash.
People looked at quite a few different answers, including copying the movies and separating them into three films, one for each color, but the problem was simply solved when people discovered you can greatly extend the life of dyes by storing them in the freezer.
You have to keep them out of light also. Conversely, high temperatures accelerate aging. I have a room in my house (an old farmhouse with uncontrolled humidity) which is very bright and somewhat intemperate (and also has a lot of indoor wildlife) and I’ve found prints made with fugitive dyes fade badly in six months — I didn’t get mad instead I use it for accelerated aging tests, I make two prints and store one in the dark and put the other in the wall.
“Handing someone a demo CD is like saying to them, Hey, you throw this away.”
(Can’t recall the comedian, could be Dimitri Martin)
It’s funny but burned CDs are a great handout for me. I can see through DistroKid that some people loaded them into iTunes and I got a small match but if money for it. Considering how cheap they are and “put in walk away repeat” the process is, I’m still happy to crank them out now and again.
Plus! I have CD labels so they have personality and flair. Also they fit a 40-60 minute DJ set perfectly. Burn, pop in the car, party on the go baby.
So much so it creeps into real life for me from time to time.
About 10 years ago working in the Comerica building in downtown Dallas (you can see it in the skyline photos) I parked in a garage connected by hallways and tunnels to stay out of the heat or rain. There were two sets of escalators in this route. One set broke. Management put up a sign on a stand stating “Escalator out of service” and within about a week somebody opened Word or PowerPoint set it to landscape, printed out “Escalator temporarily stairs, sorry for the convenience” on a piece of paper and taped it over their sign.
I’m sure I snapped a picture of it for the week or so it was left up.
> BD-R (dye or non-dye, single layer or dual layer) 5 to 10 years
> BD-R (non-dye, gold metal layer) 10 to 20 years
> BD-RE (erasable Blu-ray) 20 to 50 years
I find hard to believe that info. Seems to be sustained by a single study. That would mean that bluray (specially the BR-R format) is a pretty flawed technology.
My educated guess would be something along these two threads:
* Higher data density => tighter tolerance => same physical degradation is more catastrophic (DVD ~7x density of CD, BD-R ~20/10/5x DVD density = up to 140x CD density)
* Produced later and therefore like all consumer goods, less regard for durability and more regard for absolute cost optimization
Basically correct, with a tiny caveat: there can be manufacturing defects where the reflective aluminum layer of pressed discs is not fully sealed and oxidizes over time.
I am surprised that we're still stuck at Blu-Ray levels of data storage on disc. I would have thought we would be up to a terabyte or so by now, but everything has stagnated.
Tape remains expensive for the home data hoarder, and there is a point at which cloud storage becomes more expensive than tape.
Sony and Panasonic are making 500GB optical disks as an LTO tape alternative for businesses (See Sony’s “Optical Disc Archive” - Panasonic has a different name for the same standard).
However, these don’t really have a use for consumers. CDs had music, DVDs and Blu-rays had television and films, but there is unfortunately no compelling use case given home media is turning to streaming. Additionally, for many, cloud storage is replacing the need for local storage in spite of its drawbacks.
I'd like Microsoft Project Silica to come to the market. Basically laser burning your data onto a cube of glass, like CD-Rs it is write once read many, but it can store so, so much more, and also once the bits are 'drilled' into glass...there is nothing to delaminate or stop reflecting. Is the glass scratched? No? Then a reading laser can see the bit state.
I should check my archive, but as of years ago, I have yet to find one bad disk. Anyone who cared could easily read up about it: writing at maximum speed produces marginal media. Writing at lower speed produces media with lower initial error rate and higher contrast, so much better longevity.
That was the kind of thing that kind of just seemed logical to me back then. I don't remember reading it anywhere, I think I just had the perception from VHS tape recording lengths.
Once made a quick $5k, because a former client had started using CD-RW for their "enterprise" backups, 10-years later they had unreadable data and contacted me to see if I still had some source-code.
Checked my stack of hard-drives, and... Yes... yes, I did have a working copy.
That cemented it in my personal world - NAS with also HD's is the way to go.
What would one use as a media for long-long-term storage, taking only into account the resisiliance of the ability to read the data from it?
For example if you wanted to store data in a time capsule for like 100 or 200 and you where ganrateed the people on the other side had a reader capable of reading whatever media, and whatever format you chose.
Edit: 100 - 200 years is probably 2 to 5 pandemics, some of which may be considerably worse than the last one; and plenty of time for fires, floods, storms, earthquakes, and/or war (civil or other). Granite has a track record. Some stainless steels might be OK too (not for much longer than 200 years, though).
the library of congress uses contemporary storage media (hard drives), and migrates everything to newer storage media as the previous media becomes obsolete and replacement hardware for that media stops being manufactured.
they have learned that stockpiling old reading hardware and software isn't sufficient to be able to read that media in the future.
they move everything to new media manually and archivists and volunteers work continuously to bring data on hardware more than one generation old into the current generation as everything progresses.
source: I read it in a detailed article about the LoC a couple of years ago and have forgotten where. :(
Paper is a good one but the media is low-density and fragile. Although pre-industrial forms of paper (e.g. cotton paper, or papyrus) might have more long-term resistance than modern wood paper.
Fired clay tablets are much more resistant, but lower density (because clay), unless you use a dedicated press maybe (most of the ancient Near East clay tablets were actually fired fortuitously e.g. by a building burning down, that was only rarely intended and most tablets were used as erase-boards). But obviously non-tablet pottery can go through millennia in rather good conditions.
Carved wood is also a good idea, but it still needs good conditions and maintenance. Still, in the right storage it can last for centuries (see the Tripitaka Koreana for instance).
Leather, parchment, and vellum too, but I think then your main concern will be the degradation of the pigments.
If your focus is on longevity above all else, then you can’t beat stone. Though that can still be destroyed. I guess above stone would be highly non-reactive metals, however you’d need a metal which is not very precious as well (e.g. not gold, or silver, or platinum). I’m not very versed into metals and metallurgy, but I’d look towards titanium (it’s extremely reactive but not at normal temperatures), or bronze.
At one point there were disks made out of glass (versus the conventional polycarbonate), in theory that gives them infinite lifetime short of being physically destroyed. Though it does assume a compatible drive device.
> If your focus is on longevity above all else, then you can’t beat stone.
Stone can degrade as well. Basalt can absorb CO2 [1], marble is vulnerable to acids [2], and generally acid rain is a huge problem for historic buildings [3], as is bird poop [4].
Probably the only material impervious to damage is diamonds.
> Although pre-industrial forms of paper (e.g. cotton paper, or papyrus) might have more long-term resistance than modern wood paper.
The problem is not wood paper as such, it's production processes that leave some acidic substances in the paper, which causes it to become yellow and brittle on a timescale of a few decades.
Acid-free paper will easily last hundreds of years, and is nowadays easily available.
There is always a trade-off between mass, durability, and data density. I know you excluded those considerations, but for only 200 years in a time capsule you likely don't want to use large stone blocks or stainless steel.
The tape has several advantages. Mylar is very light and kept in the dark at a constant, reasonable temperature is going to last two centuries no problem. You can read the tape by hand, if you need. An 8-bit tape is an inch wide and can be rolled into a canister for your time capsule.
Of course, for small amounts of data print English words on acid-free paper. For large amounts, tape is going to blow out the mass budget you didn't know you had because of it's low data density. But there might be a space in there somewhere for a tech I loved as a kid.
Get a manufacturer to make you a pressed CD using glass as a base layer and gold as the reflective layer. Glass CDs are actually a thing[1], and a gold reflective layer should not complicated the production that much.
I'd actually expect such a CD to last tens of thousands of years.
For your requirement of 100 to 200 years, a pressed (not burned) CD might well be sufficient if the storage conditions are optimal (cool, dry, no oxygen).
The Long Now Foundation chose analog etched metal for a Very Long-Term backup [0]. Although their time scale is more link 10,000 years, much longer than you are thinking.
As others have noted, the answer depend a lot on whether your data size is in kilobytes, megabytes, gigabytes or terrabytes. For text (kilobytes) there are a lot more options that for video (gigabytes).
Interesting that it looks like Blu-Rays in all forms have much shorter longevity than the preceding DVD and CD formats. Presumably due to tighter tolerances required to fit more data on the same platter size?
I had switched to large format 50-100GB M-DISC BD-Rs a few years ago, but to be honest I'm often too lazy to bother with the process of burning discs for offline storage.
I should probably upgrade my NAS/external HDD&SDD stack again.
I would advise anyone going the NAS route to think carefully and read reviews on RAID devices. It's probably safer & easier to keep 2 copies in 2 formats on 2 different media in the end. I have had numerous consumer grade NAS RAID devices fail to the point I almost find it an additional failure mode rather than level of safety. Fragility of consumer grade RAID reminds me there's a reason our employers hire storage engineers!
Oh how many doubts I have... All of my CR-RWs from a decade or two ago are all unreadable. All CD-Rs from the same dates read fine. Even at the time it was known that RWs lose the data quickly. Strange to see this "20-50 years" here in this table row.
That longevity table is under archival conditions (climate controlled). The vast majority of CD-Rs I had, regardless of brand, were completely gone after 10 years of storage in an enclosed textile bag. CD-RW would barely last 2-3 years.
Brand / media quality also matters a lot. I had some cheapo bulk CD-R whose top (reflective) layer started flaking after a few years (stored in their sleeves in a cardboard box in a dry cabinet).
Stuff from reputable brands still look great, but I haven’t hard had a CD or DVD player in years if not decades at this point so it might be a cesspool of data loss for all I know.
I used to buy those thick white labelled discs in a spindle, with a green reflective layer, and remember them being quite a bit more expensive. Can’t say I I’ve ever noticed a difference.
The ones that are still playable (I hope so? Last checked maybe 5 years ago) are all industrial pressings.
As long as price per capacity / storage density increases exponentially, isn't the best solution to transfer your storage to a newer medium over and over rather than taking a bet on a long term durability of a medium that we have no way to test (and a bet on the durability and forward compatibility of the device to read it).
A single $500 20TB hard drive can contain 30,000 CD ROMS and 500 Blurays. Surely the economics are there.
Primary data aquisition aquired during the 1980s is still stored as "original" on various optical formats .. although secondary copies have been made.
Exploration | geophysical | seismic surveys that "can never be repeated" ( the atmosphere | climate | ground has changed since ) and cost thousands to millions per day to acquire have troves of original copies - SEG-9 (SEG-21) track tapes, DAT tape, optical discs, that tend to be preserved "for reference" despite much of the data having migrated media-wise forward.
It's a bit more nuanced than that. Standard CD-R and DVD-R uses organic dyes for the recording layer, which are sensitive to light and humidity. M-DISC DVD-R used some proprietary inorganic layer, which was more durable in theory and in independent lab tests[1].
When BD-R was introduced, all discs used the same inorganic recording layer, including M-DISC BD-R. That, plus the lack of independent tests, plus the fact that the manufacturer Verbatim likes to muddy the waters between M-DISC DVD-R and BD-R, led some people to conclude that there is nothing special about M-DISC BD-R. However, after a few years, disc manufacturers started making BD-Rs using organic recording layers again with inferior durability[2]. Initially these were marked as "LTH" discs (with the original non-organic BD-Rs marked as "HTL"), but nowadays they mostly stopped marking them at all so you simply don't know what you buy.
With M-DISC BD-R you'll likely get a HTL disc (but it's not garanteed, since they don't advertise it either). Additionally, there's more to durability than the recording layer (such as the material of the reflexive layer, and the overall build quality). And there is at least some anecdotal evidence that M-DISC BD-R are indeed more durable than the median BD-R[3,4].
The consensus among enthusiasts is that for BD-R, M-DISC and Panasonic HTL (Made in Japan) are very good. But they come with a big price premium (about $2 per disc vs $0.60 for some generic LTH disc).
Like also for floppy disks, the older CDs had typically a higher manufacturing quality than the more recent.
This is shown in the table from the parent article, where there are 3 rows for CD-R, the best, which were specified as suitable for archival purposes, but Kodak and the few other manufacturers of such CD-R have stopped soon their production, the second best, which were the original CD-R with phthalocyanine dye and then the other CD-R, whose expected lifetime is only half, with cheaper dyes, which were introduced on the market later.
Also, as expected, the table shows that the higher the information density is, the lower the expected lifetime is, when similar materials are used.
I think that optical disks is a lot like a Lemons market[1]. Buyers can't evaluate the quality (durability) of disks, so are not willing to pay high prices, which drives high quality suppliers out of the market, which causes buyers to offer even less, which drives even more quality suppliers out of the market, resulting in a death spiral.
Do you test the data is identical? I've got a stack of old CD-R's containing mp3 files from 20 years ago and lots have unreadable files or worse: data that reads correctly but won't play without glitches. (corrupt mp3's can produce loud spikes that are very bad for your ears and/or audio equipment)
I'm quite surprised that CD-Rs have such a long potential longevity, though I suppose they are only a certain kind (that I almost certainly have never used when I was burning CD-Rs) that are designed to be more durable. It also kind of assumes that they're keeping them in an environment suitable for archival purposes, which does not exist in my home. Still, even the lesser CD-Rs are rated for much longer than what I would have pegged them at, given just how many CD-Rs I have that have already begun deteriorating. Perhaps the ones I was buying in giant spindles weren't advisable for archival use. I've many CD-R that I or friends have burned that have their layering flaking off, making them completely useless, and I'm sure that many people out there do as well, which is why I am skeptical of using them for anything important.
In the end, for people's personal archives, they should certainly follow typical data integrity advice (i.e. the 3-2-1 rule) and also importantly, heavily curating what they choose to hold on to.
It would be really useful to have some storage media that can truly last 50 years. It is pretty disappointing that most of the media with useful capacity (dvd r etc) cap out at 20~50 years. I wouldn't trust the most commonly used storage now (usb thumb drives) with important data for even few years unless it was stored in triplicate.
Personally I think the best method of long term data storage is spinning hard drives stored disconnected. I have a collection of them that goes back decades (with important data duplicated on multiple drives) including my first ever hard drive bought in mid 90s. Of my entire collection only one HDD broke down (actually one of newer 1tb usb3 drives bought few years ago - it started clicking few weeks ago).
I can't vouch for the current spinning disc tech, but I distrust current SSD tech(SD cards and nvme included) more so my current data gets backed up to a set of spinning discs frequently.
I have a 2TB SSD in a caddy that I use for most of my stuff, then I have 2 disk RAID 1 array that is only used as periodic backup (only connected to power once a month) as well as having 2TB of cloud storage. I know I’m “good”, but it seems like there has to be a better way.
Does anyone have longevity information on commercially-pressed CDs and DVDs? I know that from my own experience they last longer, but not forever. When they were new, one of CD's major selling points was their longevity. This seems have been wildly over-estimated, however.
Not sure about the claims that blue ray's longevity may be less than cds and dvds. The technology greatly improved in the past decade (the article was written in 2010) and multiple sources saying the opposite. M-DISC brds seem to be capable of long term archiving.
Disks didn't always write a block in the correct place. Things are better now, but still not perfect.
I would recommend that you read all your data all the time; this ensures RAID-1 or ZFS will correct the silent bitrot. Hopefully. A megabyte or two per second read rate is all you need.
Recordable media is a crapshoot. Some discs are great, others are terrible. I wouldn't trust CD/DVD-R for long term media storage. These discs can also be harder on your optical drive. I've seen lots of Sega Dreamcast and Xbox consoles with prematurely worn out lasers because the owners mostly played burned games.
Pressed media has held up pretty well in my collection. I have CDs from the '80s that ripped perfectly as recently as last year. I've heard of discs with manufacturing defects that allow air to slowly reach the aluminum layer and oxidize them, but I have not seen this myself except on LaserDiscs.
I remember when CD-Rs were the savior of all our storage problems, and I weighed in heavily. They were easy to burn and very durable. BUT, just like VCR tapes were doomed because of the ease of coping video, CD's were doomed because of music and video coping. All media types are at the mercy of corporations with their own agendas. Just look a DVDs, Blu-Ray, they told us, would be better and longer lasting as far as use goes. But, really, who buys BR movies anymore?
I love and use all USB drive devises now, and SSD and HDD. But how long before the corps determine it's no longer good for them?
Nah, my personal experience told me it's highly unlikely that long. Last time I checked my CD/DVD archives of around 10 years old. Checked some random 20% of all, 50% were completely unreadable, 30% readable but with noticeable losses, other seemed to be good but still I had no confidence of the integrity of the data. So I dumped all of them, even almost of full bucket of 50 unused DVD-Rs and quite a few DVD-RWs. Since then I did not really touched any optical drives at all. To my surprise, all my data left on HDDs or Flash drives were still good without any issues.
I have binders full of cheap Staples CD-Rs from the early 2000s that literally flaked and peeled. I also have Sony CD-R discs in the exact same binders from the same time period. The Sony discs are fine.
From my experience, commercial CD/DVD seem to have way higher quality and longevity than your personal CD-R/W etc.
I've purchased lots of 80s second-hand CDs that often have awful case conditions and they all read totally fine (as in, the CRC32 checks out with online database).
Is there a reason for that? Do they simply use better material or they have better tech to write them?
Yes, the processes are very different. Commercial discs have physical pits pressed in the recording layer, while burned disks just use the laser to change the reflexity of the layer (which is some dye or phase changing alloy).
I can still read my backup CDs from 1998. I recently invested in M-disk DVDs and a compatible burner. Not cheap ($10/disk), I only use one every other month, I use the regular DVDs the rest of the time.
BTW I calculate and write MD5 checksums on the backup DVDs, so each file can be validated years from now.
You are lucky... And/or just don't have a large sample size. Of the multiple hundred CDs I've burned back in the day, some failed in just a few months. Admittedly, it was in tropical (high-humidity, high-temperature) settings.
As for HDDs, I lost about a quarter to a half, with the rest just being obsolete these days. I can definitely attest from personal experience that hard disks from the same series tend to fail at the same point (I had 3 WD blue HDDs in a single computer, as system, data, archival disks; the system one failed, so I moved the system to the data one, which failed a bit after; so I retired the third one; failures happened at about the same runtime according to SMART data).
"DVD and BD (read-only, such as a DVD or Blu-ray movie) 10 to 20 years"
This is literally unbelievable to me. I have yet to encounter or hear about a single instance of a commercial disc failure, and that goes all the way back to double-sided DVDs in the old paper cases with a clip.
I really found Dvdisaster helpful for rescuing DVDs.
The only strange thing is, that somehow the version I can currently install from my package manager appears to be missing functionality that was once available when I installed it from Debian packages a few years ago.
Does anyone have any suggestion/information on how SSD are reliable for long term archival? When they are used only once to write and then stored away?
There's no substitute for regularly copying stuff forward onto new media. This has become practical upon the advent of ever-increasing capacity disk drives.
I got to live again some >20 years old memories, pictures of younger me with my friends. stupid things you would save as a late nerd teen with access to the Internet and a ton of emails in Outlook Express format (guilty!) that I could not open yet. I guess the format is probably not super-hard to parse (most of the content is plain text actually) but since I'm here, if you know of something already existing to convert it to mbox/maildir it would be cool!
[1] https://www.gnu.org/software/ddrescue/