
One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats - thejosh
https://www.backblaze.com/blog/hard-drive-reliability-stats-q1-2016/
======
rsync
Since we're on the subject of hard drives, I hope this will be helpful or
interesting:

It has been our experience that it is no longer possible to buy new, non-
fraudulant drives of _a sort of recent, but not brand new_ drive model from
Amazon.

So, for instance, in early 2016 if you want to buy 4TB enterprise drives from
Amazon, you will find them, they will be classified as brand new, and they
will be sold by some big amazon parts seller.

When you receive them they will be nice and shiny brand new wrapped -
perfectly sealed - and when you spin them up, SMART stats will show 4000-6000
hours of use and that they are 2-3 years old.

This is almost universal and has been happening for at least 3-4 years. These
sellers are selling the drives as brand new and they are anything but. When
you complain, they will immediately exchange or refund - there's never a
hassle there - and once in a while the seller will spout some bullshit about
the drives being "new pulls" ... that is, drives they stripped from unsold
servers/desktops.

Hope that helps.

~~~
ChuckMcM
Interesting. Just before I left Google they were looking to resell "old"
drives from their machines. Basically after 3 years they had been depreciated,
so the drives would be securely erased and sent to a reseller who would sell
them (presumably marked as used) for a good price. However, those drives would
have 20,000 - 30,000 hours on them not 4,000 - 6,000. When we decommissioned a
bunch of Blekko servers the guys buying them were mostly interested in the
drives (easy to resell) but again they had a lot of hours on them. Warranty
replacements we had done over the years had a special label that said they
were refurbs.

All of that to say I'm wondering who runs a drive for 250 days and then sells
it. I could easily see a thousand hours being run as burn in to catch infant
mortality failures but those are pretty much all gone after the first thousand
hours.

~~~
wantron
At the few datacenters I worked at the drives were being crushed and put into
bins for recycling. Burn in would only take a day or so. A thousand hours is a
bit too long for a burn in cycle.

------
pazra
Backblaze is such a great company. Not only do they offer almost unlimited
backup for just $5 a month, they also publish extremely useful articles such
as this one.

Good to see Seagate has improved the quality of its product significantly, but
a 3.5% failure rate still seems rather high. How old are the drives in
question?

~~~
saryant
How's the restore process?

I've actually been backing up to Backblaze for about a year now but (knock on
wood) haven't had to restore any data. That said, one of my drives has been
acting up in the last few days so that moment may be at hand (though I also
have it backed up to a second local drive).

~~~
ansible
In addition to what Yev has said, I'd say:

You don't have backups, unless you can do restores. So you have to practice
doing restores.

For my servers at work, we're using btrfs snapshots and send/receive to the
backup host. So restoring files is just going into the appropriate snapshot
directory, and copying out the files of interest.

If your backup scheme is any more complicated than that, you need to practice
it at least a few times per year so that it is completely familiar.

Hilarious story from the old days...

We were doing backups to QIC tape drives. At one point, there was a lightning
storm. The servers were plugged into UPSs with power protection though.

However, when running a backup, I noticed that the tape drive sounded a little
different. So I check one of the backup tapes... the tape drive would no
longer switch over the tracks on the tape. So it was just overwriting the same
track again and again. Corrupted backups. Worse yet: silently corrupted
backups. No messages from the OS about a hardware problem.

That could have been bad news if it wasn't caught quickly.

~~~
existencebox
I realize this is slightly off topic, but I want to nerd out for a moment re:
your comment on hearing something wrong with the tape drive; a skill I always
felt was under-recognized for how much of a "superpower" it gives you, that
being how critical sound is for a good sysop. Broken AC belts, bad hard drive
backplanes, boot cycles, all things I've run into where the sound was the cue;
detecting an unalerted tape drive failure is the icing on the diagnostic case.

~~~
ansible
I had another incident like that earlier in my IT career.

I was a 'terminal room consultant' in college... back when we had serial
terminals hooked to Unix systems. Part of the job was the care and feeding of
a couple printers, a big ol' line printer (green bar paper) and a Printronix
graphics printer (dot matrix, for printing out fancy lab reports you wrote up
using troff).

So over time, from loading paper and clearing jams, I had accumulated hours
and hours of hearing these two guys chatter as they went about their business.

At one point, I noticed that the Printronix printer sounded funny. Just off,
in some way. So I call it in for maintenance, but they don't seem to care what
an undergraduate punk thought about printer sounds.

Sure enough, a week later, I see it is down and taken apart for repairs.

Your ears, your nose, all your senses should be used for debugging and general
investigation.

~~~
linker3000
Here in the HGST EMEA lab, you will often find an engineer listening to a
drive spin up with an induction pick-up and amplifier, muttering something
like "Yep, this one's running firmware XYZ", or "Hmm, sounds like this one has
the older, unmodified ramp".

------
osolo
I really appreciate these blog posts and have used the data they present when
buying HDDs for personal use.

One thing that bothers me is that the data presented doesn't really take into
account the age of the HDDs. For example, if a batch of HDDs of a particular
model is 6 years old and has a failure rate of 12%, that really doesn't tell
me much except that it's an old HDD.

What I'd like to know, for a given model, what the blended failure rate is
after 3mo, 6mo, 12mo, 24mo etc of operational time. That would be a real
apples-to-apples comparison.

~~~
polarix
[http://bioinformare.blogspot.com.au/2016/02/survival-
analysi...](http://bioinformare.blogspot.com.au/2016/02/survival-analysis-of-
hard-disk-drive.html) has graphs of survival rate over time broken down by
manufacturer and model. Far more informative, you're right. I don't think
Backblaze uses 6 year old drives. :-)

~~~
osolo
Thanks for sharing this. It's exactly what I was looking for.

tldr; It clearly shows that HGST has an overall superior survival rate over
time. WD is a distant second and Seagate in third (although the Seagate
ST4000DM000 model is exceptional and fairs very well).

------
xhrpost
Wow, did I read that right? They are filling 3+ vaults per month? A vault is
20 pods. Each pod is 480TB. They've mentioned in the past that they use 17+2
redundancy. So each pod should be about 429TB of added storage. At 60+ pods
per month, they're adding 25,740TB per month?!

~~~
atYevP
Yev from Backblaze ->

That's about right ;)

------
tibbon
I wish some other companies that use _massive_ amounts of hardware (Dropbox,
Amazon, Google, Facebook, NSA) would release data like this. I love that
Backblaze does it, but having an even larger sampling of data would be great.

~~~
brianwski
Brian from Backblaze here. It completely, utterly blows my mind that we are
the first and still only company to do this.

First of all, it is a great way for people to hear about our company, which
sometimes leads to people buying our products and services.

Second, even just for the good of all humanity, why wouldn't you do this to
promote the best drive manufacturers and heck, even help them learn how their
drives are working in the field?

It boggles my mind that Facebook, Yahoo, Google, and Amazon (Amazon Web
Services include S3) don't publish their drive failure stats. What could
possibly be the reason they keep silent?

~~~
hinkley
You guys are in the Do Well by Doing Good category on this, and unfortunately
few companies in any business sector maintain this philosophy as they grow.

The boardroom is so divorced from the showroom that it's not even funny how
big he disconnects can get.

------
anexprogrammer
Interesting that HGST do so well, yet are now part of WD, who do so badly.

I would have expected more absorbtion+rationalisation from the takeover, yet
they still clearly look of the IBM family than anything in WD's ranges.

IBM always had a pretty good rep for failure rates, aside from a couple of
horror drives in the 90s. I wonder how they've managed to keep their rates
markedly better than other makes, even under changes of ownership.

More to the point I wonder why WD haven't been able to improve rates as a
result of taking over IBM/HGST.

~~~
atYevP
Yev here -> From what we can tell and have been told they are run as
completely different entities. Hopefully that quality spillover will indeed
happen in the future, but for what it's worth WD isn't doing all too poorly,
they just tend to be slightly more expensive for us than the Seagates at the
moment.

------
Ologn
Having had many years of experience as a sysadmin, I have one word for those
concerned with hard drive reliability - temperature. Heat is the enemy of hard
drives (and other system components).

Sometimes heat causes a machine room meltdown, where machines start failing
left and right. This is not uncommon. What usually happens is this - the
company knows enough to put their web server etc. at a controlled colocation
facility (which isn't always ideally controlled, but that's another story).
But the developers would like a local file server, development source control
server etc. There is a spare, windowless room in the office, and without much
planning, a machine or two goes in. Then a machine goes in attached to a tape
drive, which backs up those machines and the local desktops. Then more
machines go in, then more. All these machines generate heat, so a room cooler
is bought. But it drops condensation in a cup, and shuts off when the cup is
full of water. So someone's job becomes to empty that cup every morning. But
then summer comes, and on a particularly hot Sunday afternoon the condensation
cup fills, the cooler shuts down, the outside temperature combines with the
temperature of a closed, windowless room full of machines for hour after hour.
Finally one machine has a component fail. Then another. Then e-mails and phone
calls and panic starts flying.

I have seen this happen more than once, and have heard about it more. It
always starts as an ad-hoc, unplanned, "temporary" solution for "unimportant"
machines. But as time goes on, and machines are added, and business
dependencies are formed, you have to start supporting a machine room that was
never planned as a machine room.

------
taspeotis
> For Q1 2016 we are reporting on 61,590 operational hard drives ... [t]hese
> days we need to purchase drives in reasonably large quantities, 5,000 to
> 10,000 at a time.

To me, that's an astoundingly large number of hard drives. But I realise there
are probably much bigger deployments. Does anybody happen to know just how
many hard drives Amazon or Microsoft have for AWS/Azure?

~~~
alecbaldwinlol
I imagine maybe 10x more than this. The numbers actually aren't too shocking
if you think of the mirroring of data that's required for maximum uptime (not
even archival), and the occasional disk failure.

1,000 TB (1PB) can be easily handled across ~150 (6-7TB)HDDs for one copy, but
300-450 HDDs would be required for additional mirroring.

Largest tape cartridges out there are between 6 and 8.5 TBs, and cost around
$22 per TB. That's only $22,000 per PB, and this is for high throughput
cartridges like LTO7 or StorageTek Titanium. LTO5 is much cheaper.

Considering that the largest tech companies and major organizations routinely
cut POs for several $100ks and are dealing with 100s of PBs of data across
disk, tape, DVDs etc, it isn't outside the realm of possibility to have
300,000+ individual disks and tapes floating out there :)

~~~
jdcarr
Anecdotally, Google's file system Colossus uses Reed-Solomon 1.5x replication.
So those 150 drives might only turn out to be in the low 200's.

And I remember reading a tweet from a Google engineer that they would be paged
if their free storage dropped below 5PB.

~~~
raisedbyninjas
I thought waking up at 3am, shambling to my desk and connecting to the VPN was
bad. Imagine having to drive down to the datacenter and rack 200 hard drives.

------
Veratyr
So my math is probably messed up here but:

\- Price of buying a drive and a pod to house it: ~$0.036/GB [0]

\- 1.84% of drives need replacing each year (warranty aside)

\- They use 17:3 parity for redundancy (15% of storage)

So the hardware price of a GB should be something like $0.036 * (1/0.85) *
1.0184^(years). For 10 years, the hardware would cost $0.05/GB, or
$0.0004/GB/month.

Power costs and DC space of course need to be taken into account but I still
find it interesting that the hardware itself costs only ~10% of what they
charge for B2.

[0]: [https://www.backblaze.com/blog/open-source-data-storage-
serv...](https://www.backblaze.com/blog/open-source-data-storage-server/)

~~~
askldfhjkasfhd
How did you choose 10 years?

~~~
Veratyr
Arbitrarily. I suspect after that point another storage technology will
supersede hard drives.

------
rdl
From reading this, HGST >>> Seagate, right? I wasn't sure how the "weird"
sourcing they had (cracking external drives open during the HDD drought) would
affect things, but that was over in 2015, and the trend still seems to be HGST
> Seagate.

~~~
wiredfool
Seagate had one horrifically bad, almost guaranteed to fail drive in the
sample, the 3TB ST3000DM001. Many many people lost data when they died
(according to reviews @ newegg and amazon).

~~~
rawnlq
It is currently the best gb/dollar drive right now at $70 so I was tempted to
get it but then saw the backblaze article.

People on /r/buildapc don't seem to believe it's a problem though:
[https://www.reddit.com/r/buildapc/comments/4jkpor/avoid_the_...](https://www.reddit.com/r/buildapc/comments/4jkpor/avoid_the_seagate_barracuda_3tb_35_7200rpm/)

~~~
wiredfool
Well, they're entitled to their opinion. It's possible that the problem was
fixed and that there are new drives with that model number that don't have the
bug.

I personally wouldn't use one for anything more critical than a door stop.
Been there, done that, lost some data.

~~~
jandrese
Yeah, I personally wouldn't buy one at this point. It looks a lot like some
sort of design or chronic manufacturing flaw. Seagate's version of the IBM
Deathstar.

------
arca_vorago
I love the work Backblaze does, and especially how much data they publish. Not
only that, but their openness of their storage pod design helped me more fully
understand the current limitations of backplanes as we move to more and more
data. I have dreams of m.2 backplanes so I can skip the limitations of sata...
but I digress.

Thanks Yev and everyone else there, a shining example of how to build a
company and a reputation at the same time. Keep it up.

------
jokoon
My 5 year old hard drive has been vibrating like crazy this year, also making
the metal case panes vibrate, which makes a ton of noise. So I searched
online, and found a solution where the HDD is suspended with cable ties, a
little like this
[http://i.imgur.com/4MXB1IG.jpg](http://i.imgur.com/4MXB1IG.jpg)

I guess I should just buy another one. I'm going to lose all my data anyday
now.

~~~
jerf
You're going to lose your drive any day now. Not necessarily because your
drive is vibrating... just because you're _always_ going to lose your drive
any day now. Anything on only one drive can be on only zero drives in the
blink of an eye. If that's a problem for you, best deal with it before it's a
problem.

It is true that hard drives _often_ give a surprising amount of warning. I
don't think I've ever had a drive totally spontaneously fail on me. But it's
still best not to count on that.

~~~
jandrese
I once had a drive that somehow managed to corrupt its own firmware. Total
garbage on bootup (nonsensical LBA numbers, returned model number was garbage,
SMART info was totally corrupted, etc...).

~~~
sounds
Hard drives store firmware (executable code) on hidden sectors of the drive.
The amount of firmware they store in flash is relatively small.

That may help explain how your drive's firmware got corrupted.

------
ck2
That lack of failure on the Seagate 6TB is impossibly impressive for Seagate.

Someone else must be making the drive for them or they have a new factory
doing something different.

------
redcalx
Wondering if data centres typically power down a drive after some period of no
accesses? Seems like you could save a lot of power and gain some HD life that
way - so long as a drive isn't frequently spinning up and down. Seems like the
data access patterns could be monitored and less frequently data moved to a
'low access frequency' HD bank(?)

~~~
atYevP
Yev from Backblaze here ->

For us the drives are constantly spinning, we don't power them down. One of
the reasons is we never know when a customer will want a restore, so we have
the data available 24/7\. That said with our Vaults
([https://www.backblaze.com/blog/vault-cloud-storage-
architect...](https://www.backblaze.com/blog/vault-cloud-storage-
architecture/)) it's theoretically possible to power down entire cabinets and
still have the data available, but we don't currently see a need to do that.

~~~
redcalx
See the other comment: there is a lot of data that isn't accessed for years at
a time, so you're essentially spinning disks for years to save a few seconds
of spin up time.

I think he's right - energy is too cheap for this to be a commercially
relevant factor to you, but if the true cost of energy was factored in
(climate change, the human cost s of the relatively dangerous mining and oil
drilling industries, local pollution, etc) then I think that would tip the
balance.

Oh well. OK, final suggestion, maybe do it and spin it as a public relations
win? :)

~~~
atYevP
With Backblaze B2 the data has to be available at all times, so powering them
down isn't really feasible. For personal backup it might have been, though the
manpower it would have taken might have also knocked it out of whack a bit.

------
heywire
I'd really like to try Backblaze, but I use a Windows Home Server 2011 Atom-
based system as a makeshift NAS to store all of our family photos. The
Backblaze Personal Backup installer fails with an error because it is
considered a "Server Operating System". I'm currently using CrashPlan with
this system.

------
35bge57dtjku
> For WDC, we sometimes get offered a good price for the quantities we need,
> but before the deal gets done something goes sideways and the deal doesn’t
> happen.

Sound pretty interesting. What's really going on?

------
ausjke
Java is responsible for 91%* of security attacks. Backblaze's code is native
to Mac and PC and doesn't use Java. -- is this real?

Anyway I'm going to be a backblaze customer for $5 a month!

~~~
krzyk
Isn't it for the applets?

------
urza
I wonder, what filesystem are they using. Or more general the whole software
stack (what OS, etc) would be interesting to know.

~~~
urza
Ah I found it on their blog: [https://www.backblaze.com/blog/vault-cloud-
storage-architect...](https://www.backblaze.com/blog/vault-cloud-storage-
architecture/)

They use ext4:

 _Each of the drives in a Vault has a standard Linux file system, ext4, on it.
This is where the shards are stored. There are fancier file systems out there,
but we don’t need them for Vaults. All that is needed is a way to write files
to disk, and read them back. Ext4 is good at handling power failure on a
single drive cleanly, without losing any files. It’s also good at storing lots
of files on a single drive, and providing efficient access to them._

 _Compared to a conventional RAID, we have swapped the layers here by putting
the file systems under the replication. Usually, RAID puts the file system on
top of the replication, which means that a file system corruption can lose
data. With the file system below the replication, a Vault can recover from a
file system corruption, because it can lose at most one shard of each file._

------
forgotpwtomain
There seems to be an incomparably tiny sample size for about half of these...
1 out of 47 drives gives a 8.63% annual failure rate.

~~~
pgrote
Yes, they mention this in the article.

"Failure rates with a small number of failures can be misleading. For example,
the 8.65% failure rate of the Toshiba 3TB drives is based on one failure.
That’s not enough data to make a decision."

