Let's take a typical small business system's requirements. We have a 2U database with 12 3.5" hot-swap disks, 2 200GB SSDs for caching and 10 3TB disks in a RAID10. We have up to 15TB that we want to archive.
Optimism: we get 100MB/s write speed and a write BER of 10^-15. We swap 30 500GB disks, each taking about an hour and a half to write, and about one in a thousand disks has an unrecoverable error.
More likely: we get 75MB/s sustained write speed and a write BER of 10^-14. We swap 30 500GB disks, each taking close to two hours, and we can expect one in three complete sets to have an unrecoverable error.
Press releases: not quite useful.
At about a quarter of the price of the LTO tape drive alone -- data can be recovered anywhere for the next 20 years or so (I have tapes from 2002 that I was unable to find a tape drive for in 2007, but I have a pair of 15GB IDE disks from 2000 that I checked last month and are still readable).
At $60-$80/2.5TB, ($24/TB) LTO6 does have an advantage to disk ($40/TB) if you have hundreds of TBs to backup. However, because of the cost of the drive itself ($3000), you only break even after 200TB or so. And if your drive breaks (or you need to backup in one place and recover in another, as is often the case with DR plans), you break even only after 400TB or so.
Since approximately 1998, hard drives for backup beat tapes on capacity, price, access speed (drives are random access!) accessibility (an external universal SATA/IDE can be bought for $30 or so, but you already have the connections on your motherboard so you don't even need it). I haven't been able to find anyone with a convincing argument for tapes - maybe you have one?
Hard drives are much less reliable than tapes; hard drives aren't made to be stored on shelves; and when a hard drive fails, you most of the time lose most or all of its data, while a tape failure generally affects only one file (if it's a reasonably large file and you aren't using compression).
As soon as your storage and archival needs are more than a few disk drives, do yourself a service and switch either to RDX or to tape. Storing basic HDD on shelves is a recipe for data loss. I know it because we sell backup solutions to people doing it and losing data all the time :)
I can back this up. For some reason, someone decided to back things up to hard drives and even though they are stored in anti-static bags in a 600lb fire safe, when I remove the drives from 1 year ago to re-write with that months backup, I've had up to 1/2 the drives not spin back up. Ugh.
What makes RDX (basically, an HDD encapsulated in a rubber cartridge) more reliable than e.g. an off-the-shelf rubber-USB-enclosed HDD? It's not like cartridge makers are making hard drives themselves - they source them from the same manufacturers as everyone else.
All the complaints about HDDs for backup/archive apply to RDX, as far as I can tell. And at $100-$150/TB (and I can't even find cartridges bigger than 1.5TB for sale anywhere), it seems to combine with the worst price as well.
If your archival media is going through thousands of insertions, then it's not archival - it's the most inefficient swap device ever.
But I've connected/disconnected my laptop's USB mouse and keyboard a couple of thousands of time (several times a day over the last few years) and neither connector nor socket seem to be suffering.
Furthermore, most of my cold store drives are of the self-powered USB 5400 RPM variety (getting a decent 30-40MB/sec on USB3) - if something breaks, I replace the $1 cable that connects drive to computer.
Seriously, RDX seems like an awful choice - does someone here actually know differently?
USB2.0 Full Size: 1,500 mating cycles, gold flash plating.
USB2.0 Mini: 5,000 mating cycles, 15µ” gold plating.
USB2.0 Micro: 10,000 mating cycles, 30µ” gold plating.
USB3.0 Full Size: 5,000 mating cycles, 30µ” gold plating.
USB 3.0 Micro: 10,000 mating cycles, 30µ” gold plating.
Why Micro types offer better durability?
Accomplished by moving leaf-spring from the PCB receptacle to plug, the most-stressed part is now on the cable side of the connection.
Inexpensive cable bears most wear instead of the µUSB device.
Plus as you note, it's easy to make it a cheap replaceable part. Use a USB extension cable on the computer side and you won't be wearing that out (I've done that trick in times past).
> Well I've recently restored 150 LTO-2 tapes from 2002/2003 using a perfectly current LTO-4 drive, and I'm pretty sure you'd have a heck of a hard time successfully reading all of a bunch of 150 hard drives of the same vintage without losing a single file.
That might be true. However, I have so far had perfect success with about 50 or so drives that I've restored from (across 10 years or so), and abysmal success with tapes (about 1/10 in the mainframe days would not restore).
> Hard drives are much less reliable than tapes; hard drives aren't made to be stored on shelves;
That's true, but neither are tapes. Hard drives are less tolerant to environmental conditions than tapes, they survive better when exposed to heat, and get damaged more quickly when exposed to cold (oils used to keep it running smooth tend to congeal irreversibly when exposed to cold for long). Also, at least in the past, even weak magnetic fields wreaked havoc on tapes, and even strong ones spared drives.
> As soon as your storage and archival needs are more than a few disk drives, do yourself a service and switch either to RDX or to tape. Storing basic HDD on shelves is a recipe for data loss. I know it because we sell backup solutions to people doing it and losing data all the time :)
In my current top-secret venture, I need to keep everything (~20TB/customer) randomly accessible for 3 years, and cold-store accessible or 7. So far, we just keep multiple copies (some online and some offline), and keep refershing to newer larger drives every couple of years. When I specced it 3 years ago, tapes were twice as expensive (I need a tape drive at every customer location + at least two at my office) and didn't properly address the random access aspect.
But I keep re-evaluating. So far, the next step seems to be going to a BackBlaze style pod - which would also address the valid drive-fails-completely concern you raised.
p.s., I find the "compressed" statistics really misleading. Every kind of backup software can compress - if I saw a hard drive manufacturer sell a ("2TB drive (1TB uncompressed)" I would consider it fraudulent, but that's the standard for tapes. For the record, my data is (as far as tape drives are concerned) not compressible at all -- think audio, video, jpegs, and other analog recordings.
EDIT: p.p.s: Of course, don't keep hard drives on shelves. Nor tapes. Humidity and temperature need to be controlled, and the environment needs to be anti-static. For archival purposes, the 5400-RPM usb self-powered disks are a little slow, but keep very well, in my experience, with reasonable temp/humidity.
EDIT2: (bad) memories are coming back. The biggest problem I recall with tapes in the mainframe days, were that some tapes (the daily and weekly backup sets) were continuously reused, to the point that mechanical wear and tear was a bigger problem than magnetic wear. And the even bigger problem was that you only found that out when trying to restore. One advantage of disks is that you keep reading them while writing them - wear is apparent much more quickly. If you're only doing archives and never rewrite tapes, that doesn't matter. If you keep reusing the same tapes/disks, it makes a huge difference in reliability.
I switched from USB 2 external hard drives to LTO-4 (see my other comment in this thread for one reason for that generation, which BTW per the LTO spec can be read by an LTO-6 drive) because I wanted multiple copies of my data in a less physically fragile form than hard drives (something I'm willing to routinely schlep to a safe deposit box). And I'm an old timer, bought my first DECtape in 1978 (sic), because it was neat, and then it became handy when I learned the -rf flags to the rm command ^_^. I'm very used to tapes, how they work and don't, what you can and can't do with them. And while I was flirting with hard disk backup almost had a catastrophic loss (http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/) only redeemed by luck and offsite storage to rsync.net in Denver.
I didn't buy a high end, super fast tape drive, rather an internal HP 1760 EH919 drive, Newegg will sell you one at 2,130, but with some bargain hunting you can get for $1,400 easily (we managed $1,200), and Fujifilm and HP tapes (wouldn't trust any others, except maybe IBM but those are expensive) from them are going for $39 to $42 quantity one, you can of course get better pricing if you buy packs or 20 (or more) and/or go bargain hunting, e.g. 35.45 for the first Amazon Merchant I found with a rating >= 96% for a 20 pack of HP tapes. That's for 800GB uncompressed storage per tape.
Per my other comment I also had to buy a 15K SAS drive, that was $250 back then (late summer of 2011).
And I should note one of the biggest use cases for LTO drives is fast backups for businesses. With of course plenty of $$$ backing hardware to keep LTO-5 or LTO-6 drives happy.
You still need drive redundancy in most setups. And my experience with tapes has been that they are not significantly more robust than drives: they break and die when they fall on a hard floor, they get ruined if heated too much.
And an issue that might be gone in the recent LTO era (has it?), but which I met several times: if you need to recover tapes made 5 years ago, finding a usable tape drive for use or sale might be a big problem. The last drive I had I can't connect to a modern motherboard was an MFM controller-RLL drive pair I had in 1987. IDE came shortly after.
Any sane archiving plan will read old tapes "before it's too late" (yes, I know, I know) and re-archive them. I suspect a prime use case for this proposed optical media is archiving what Hollywood et. al. does, which is getting to be rather large (10s or 100 TB per a major film?), put them in controlled environment facilities and then forget about them unless the film is such a success you want to make a new release that needs to go back to the masters, or more material for a "Director's Cut" etc.
That's true. In fact, any sane archiving/backup plan REQUIRES that you regularly verify the integrity of the data, preferably by restoring -- and refreshing that data onto new media "before it's too late". That is true for both tapes and drives.
Lifetime of tapes may not be what you think it is.
I know from experience that people are very likely to think "Tape?! It'll last forever!...right?" which is dangerous.
M-Disk 1000 years, same storage idea. $5 a disc.
Sony and Panasonic need to do something with their lead in optical disk technology, but there's no demand for a Blu-ray successor. Heck, there isn't even much demand for Blu-ray. So archiving is the way to go.
I just hope they make those things to last.
The idea was pretty good, the industrial design was lackluster but serviceable. But the longevity of the discs was questionable (especially if you took advantage of the ability to remove the spindles holding them), and the devices themselves were abruptly pulled from the market around 2011.
The drives inside were actually laptop drives--the kind you used to slide into your laptop before you had a MacBook. I doubt they would last very long, but we didn't really get to find out, because once new units were no longer being sold we had to abandon the system.
So, do not buy this thing for long-term storage, because one of its main backers already proved they do not have what it takes to service this market.
If there's any repression in the movie market, I would blame prices/multi-format bundling. I don't want to buy a six-version pack with two different digital versions, a Blu-ray copy, a DVD copy, a 4:3 DVD copy, and a VHS copy for $40. I want ONE DISC, Blu-ray, for $8-10
I'll pay a couple extra dollars for the BR + DVD bundles (Although, all my stuff plays BR these days)... I think it's fairly standard to get Widescreen these days as well. I don't remember any recent 4:3 releases for newer titles.
The main issue I have is "release day" discounts of $~10-15... followed by months of $~20-25. I hate missing a "must have" and then waiting until the price is reasonable again.
I don't mind the 2-disc BR/DVD bundles, I just wish I had the option to buy a 1-disc BR instead. It gets even harder when you want a 1-disc 3D BR.
This tells you the item was not a "must have" then, or you would have bought it already, rather than waiting...
personally, i know not one single person who still buys physical video discs anymore. personally, i can say that demand for blu-ray is non-existant.
what would be useful, here, would be some sort of data that compares usage/adoption rates for units of time after a formats release. how many people are using blu-ray X years after it became available, compared to DVD, etc.
i would expect it to not be as popular as DVD or VHS before it, but expectations and anecdata aren't terribly useful.
Not necessarily. With photography having been digital for about a decade and home video for about half that, tech savvy families easily have several TBs of data that they would want to hold on to for the long haul. The prospect of being able to burn a complete copy of these things on a few discs and leave them at the grandparents or in a bank box seems very appealing, and only more so over the coming years.
And you're of course right about the cloud, especially in the US; I've got rock solid AT&T ADSL, but it's very A(symmetric) and has a 150 GiB monthly cap, so only my very most important stuff gets saved to rsync.net, which e.g. saved my email from the tornado.
Archival hard drives exist of course, but are at a cost disadvantage compared to tapes or optical media.
SSDs are way too young to depend upon, as far as I'm concerned. For that matter, if you really screw up and have to recover data from a drive, they're much more opaque than hard disks.
Back in 1998, I dealt with HP jukeboxes the size of a wardrobe with robotic arms to pluck 9.1GB magneto-optical disks out of the racks and stick them into drives. IIRC, HP guaranteed their disks for 100 years.
I played the MO game in the '90s and found them to be rock solid, and durable as well (e.g. I'd throw a disk cartridge in my backpack without problems). Theoretically they looked quite quite solid.
Am I crazy?
No mention in the new release how long they are meant to last though. This seems like a pretty key point to me.
I would imagine that a silo with multiple robots to place disks in many readers would be significantly better than tape in most ways, and would retain the low power advantage that tape enjoys over powered systems.
BD-RE advertised 10,000 overwrite cycles http://docs-europe.electrocomponents.com/webdocs/0bdc/090076...
LTO-5 gets only 200 full-capacity writes http://support2.imation.com/downloads/imn/LTO/Usage_Life_Ima...
Edit: This one says 30 years for LTO-5 http://www8.hp.com/us/en/products/storage-media/product-deta...
In conclusion; I think that just enforcing high manufacturing standards won't suffice, there must be some material definition as well. Looking forward for more information on this.
But what happened to the promised sapphire (alumina) coating on disks that makes them scratch proof?
All of this led to the oxidation of the metal layer; rendering it unusable.
I'm thinking about services like Amazon S3 and Glacier here, together with whatever competition appears. I presume that at this level, what matters is exabyte level storage hardware (perhaps a robot-managed room of archival discs or tapes). One where the provider can switch technology every few years, and customers never have to notice.
At capacities on the level of terabytes, we're getting to the point where upload bandwidth isn't so much of a bottleneck any more, aren't we?
So no, I don't think this is going to battle storage providers.
Businesses have a choice: they can invest in buying storage hardware for a bunch of their computers every few years, or they can invest in a fast-enough Internet connection and outsource the storage. That Internet connection will bring them additional benefits over just storage.
I claim that the economics are switching in favor of the Internet option, and will continue to do so over the next decade.
There's is also the "political" problem (as outlined by everyone else), I can't see every company wanting to outsource the storage of their secret data. Then you're also at the mercy of Amazon, that for whatever reason can stop providing you service (It's a tiny possibility, but it's still there).
You know that you are talking about the typical speed of many industrialized countries, like Germany, Spain, France? I doubt those whole countries are not the target market.
We use expensive 500gb tapes to take weekly backups of our entire site which are kept for 1 year. I'm not sure how many times the tapes can be overwritten but there would be a price-point for these archive disks that would make the tapes obsolete. It would also save a lot of space and therefore money as we are charged to store the bulky tapes off site.
Small businesses still generate data that needs to be archived. Uploading a TiB of data at ~40KiB/s would take the best part of a year.
A reliable fast connection is still very expensive, and there are several other considerations.
When we looked at offsite backup a major consideration was restore times. Even if you can do differential backups, in a disaster you might need to get everything back ASAP and even the best internet connection can't get near the throughput of onsite disks/tapes.
This technology is obviously targeting this kind of customer, and I expect it to do rather well in that niche. Providing a cloud-like backup infrastructure internally is a significant cost, is rather hard to get right and requires a lot of maintenance over time.
Depends on the from and to. 1TB at a sustained 100mbs is roughly 24 hours. Or 10 days at 10mbs sustained. If your data is actually written to disk somewhere, I doubt you can sustain more than 200mbps for most services?
So, if you record home video in 4k -- you're going to reach a practical bw ceiling pretty soon. In general disk usage is going up faster than bandwidth -- so archive media isn't likely to go anywhere.
As for Amazon Glacier, cost will come to 120-133 USD per Terabyte/year -- with an additional cost of ~300 to retrieve a Terabyte in 24 hours. For comparison, a 4TB usb3 external drive costs about 150 USD now -- and you'd break even buying 3 each year -- in terms of cost/TB (assuming you could store them for free, and spent the 300 on retrieval as needed).
You can definitely beat AWS on price but there's a non-trivial up front investment required unless you're comfortable skimping on the operational benefits.
1. http://aws.amazon.com/glacier/ promises 99.999999999%
For that matter, if they were really marketing at those who are professionally concerned with long-term reliable storage (archivists), even the marketing would include some information on what makes this new media any more reliable over the long-term than existing optical media. The press release includes nothing on this, odd for something branded as 'archival'.
It looks like it's just a larger capacity optical disc (which I'm sure there's a market for), with the 'archival' part just being marketting (odd; apparently they think there's a market for this too, even when it's just spin).
But yeah, the long-term reliability of media is often a key consideration in archival storage, which is why I think it's suspicious that they brand this media as 'archival' without providing any information at all about what they think it's long-term reliability is, compared to other (optical or other) media.
For a small user, you have the extra concern of betting large on a single production run from a single vendor. If you care about archival, you'd really want to burn multiple copies using discs from different production runs and, preferably, different manufacturers.
(This is, of course, why I'd really recommend using CD-Rs only in conjunction with a different technology such as AWS Glacier which has a completely different set of failure modes)
You are of course correct about the precautions that have to be used, but there's screw cases for everything you can use to back up your data. I was merely responding to the assertion that CD-Rs are entirely unsuited for archival purposes, and backed it up with my own experiences over a decade and a half.
Me, I live a hair's breath from Tornado Alley, and use multiple means to backup my data, local discs, disks in another room (which got trashed enough to be unreliable in a tornado http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/) which I've switched to LTO-4 tape, some of which live in a safe deposit box, and the really important stuff offsite to rsync.net (which saved the only important data totally lost locally from the tornado, or at least without $$$ to a data recovery firm). And of course the CD-Rs, which at some point I'll start putting into the disk and LTO bases system.
The experienced storage admins I know share that hope but don't trust vendors not to get it wrong. It's just too easy to miss a factor which turns out to matter.
> I was merely responding to the assertion that CD-Rs are entirely unsuited for archival purposes, and backed it up with my own experiences over a decade and a half.
Question: have you done bit-level checksum validation on that old media or is that just the ability to read without errors? There's a little bit of error correction built into the format but I wouldn't trust it for anything important.
However my words are coming across in this discussion, I'm really not a very trusting guy! Many of my early computer experiences were in the early '80s with surplus hardware from the '70s, including a PDP-11/45 that was a bit beyond the 300th DEC manufactured. Of course everyone used magtape back then for backups, and those were quite reliable (originally intended to be reliable off-line storage).
ADDED: and there are other precautions to take. E.g. I only bought my optical media in April and October, to minimize the environmental stress during shipping.
In the digital preservation community, the standard is at least MD-5 / SHA-1 and most people are moving to SHA-256/512. With a CRC-32 check you're likely to get false-negatives for modern data volumes and there are disturbing reports of CRC failures at higher than expected rates which suggest that the best answer is using multiple, cryptographic hashes particularly since the computation has effectively been free for a least a decade except on unusually CPU-starved storage hosts.
1. I don't recall the paper but I believe it was a followup to http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigco...
Wow, if so, it sure seems sketchy to brand it as 'archival'.
In this context, long term durability has more to do with chemical attack from the atmosphere than being handled by human hands.
Having media that lasts for 100 years is little comfort either. What good would do floppies that last for 100 years when the only computer capable of using them here in my house is an Amiga 600?
Odd. To me, something labelled "archival" means "is expected to last a really long time, and be especially robust."
I _think_ that's what it means to most people. Although it's hard to say. But compare to wikipedia says "Archival paper is an especially permanent, durable acid-free paper." Especially durable and permanent, not especially fragile and needs to be treated with unusual care!
If they are marketting something as "archival", I am reasonably certain that _many_ people are going to think that means "especially durable and permanent", not "especially fragile, must be treated with care."
But yes, there are many more factors to long-term preservation of digital content than simply media reliability, including inter-operability with future hardward/software. Integrity of the bits on your media is certainly one factor though. And yes, multiple copies and regular integrity checks is crucial, not just counting on your media to last. Media long-term reliability still matters though.
(I work in libraries, and I work with and next to people concerned with digital preservation, although I don't work on long-term digital preservation myself).
No. I just suppose that packing that much information so tightly, encoded in such small features on a plastic disc, will make the recording more fragile. It's like printing all your text with a 3-point font. You can print a lot of text on a single page, but any damage to the page may make large parts unreadable.
As for long term preservation, I'd never suggest keeping your data on the same medium for any significantly long time. And never, ever, under no circumstances, find yourself with a single copy of anything.
I would have imagined some special material that can last longer and is at lower risk from scratches...
So it's a new standard with tougher physical requirements. Actually in a japanese press release  they present at the end the current archival solution offered by Sony, which consist of a set of 12 optical discs in a cartridge. I'd image this new disc standard could be sealed as well for better protection.
Seriously, why hasn't anyone made diskette-style discs a common standard? I have never seen a cd last more than a few years, or one single touch with a finger on the shiny side.
On the other hand, I've yet to have a Blu-ray fail, despite treating some of them rather poorly. This is just with factory-pressed discs though, I have yet to burn one. Has anyone else here had more experience with BD-Rs?
which use more advanced coating such as Durabis http://en.wikipedia.org/wiki/Durabis
(Well, unless you want to use one of the exotic technologies that etch your data on metal plates or whatever; but in practice, paper is a lot cheaper.)
That was alright when I just needed to move data around, or wanted to watch a movie once or twice, but I can't use something that unreliable for archiving.
I've never had an unrecoverable error with Taiyo Yuden CD-Rs. I just recently pulled 8 out of cold storage with no problems aside from one marginal DVD-R drive, that were burned from 7/13/04 to various dates in 2008. If you want I can check from earlier in the decade, although my first batch of discs were Kodak Gold. I systematically burned and carefully stored about 4,000 during the last decade so I've got a lot to choose from ^_^.
That's not really a great endorsement.
OK, the blanket "I've never had an unrecoverable error" is not true if include "on every drive I've tried them on", but then again this is the only drive that's ever started failing on me before it was replaced in the normal course of events. So my success rate in getting my data back without serious recovery efforts, over 14 years minus 5 months, is 100%. But I should go check some of those oldest disks again, it's been a year or two since I've retrieved stuff from the very early oughts.
I don't consider this to be a big thing, given these are commodity consumer drives which will continue to be manufactured to read CDs for the foreseeable future, especially compared to tape, where bad drives are known to eat them and they don't have one old format frozen in time (granted, CD-Rs are less reflective than pressed CDs)....
Just a few weeks ago I actually took my old spindles of CD-Rs and DVD-Rs out of storage and moved into Amazon Glacier. The disks had dates from 1999-2007. My results were that out of 80 or so discs, 5 of them had unrecoverable errors (thank god for dd_rescue).
These were burned in my youth, so the method I picked disc brands was "the cheapest disks that aren't those uncoated foil noname ones"
If i cant afford to press the disks how should i backup my data?
Right now we are using nas systems with a raid setup
Per sspiff, you do need to protect optical media from environmental extremes. Tape cartridges are more physically fragile (although I'm sure there's data recovery firms that'll repack the tape into a new cartridge) and also shouldn't get too hot or humid. And ideally are stacked on edge.
A RAID NAS is a good medium-term solution, and perhaps creating offline backups on disconnected drives every 5 years or so can solve long term storage, though it does require maintenance over time (refreshing the drives). If you are not afraid to use cloud storage, you might use those services, but I don't know how long those will be around.
Granted, I haven't looked for years at what's been done to address this, but based on the state of the art back then I'd only try this with ZFS, which checksums everything it puts to disk, which among other things catches the incredible screw case of the very complicated firmware writing the right data correctly to the wrong location on the disk (!).
On the other hand, https://news.ycombinator.com/item?id=7372595 indicates Panasonic is not trustworthy in this area.
Maybe give this 5-10 years and see if has staying power, adoption by others, etc. The LTO ecosystem shows this sort of thing can be done.
> Standards for 100 GB read-only holographic discs and 200 GB recordable cartridges were published by ECMA in 2007, but no holographic disc product has appeared in the market.
(Can somebody with more karma please flay, er, flag'im?)