
Backblaze Hard Drive Stats Q2 2020 - caution
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/
======
andy4blaze
Andy at Backblaze here. To put a pin in it, the three main factors for which
drives we use are cost, availability and reliability. We have control over
reliability as our systems are designed to deal with drive failure. That
leaves the market to decide on cost and availability. Assuming a competitive
market we can buy the drives that optimize those factors.

~~~
gentleman11
My main concern with third party backups is the privacy aspect. I never want
to worry that some silent TOS change means that ad companies are scanning all
my documents. Does backboaze have any options for e2e encryption or some sort
of iron clad privacy policy without a “we can change it at any time” clause?

~~~
bronco21016
Several backup tools have encryption in place so that your data can be
encrypted before it leaves your device. Rclone for example has encryption and
Backblaze B2 capability.

~~~
gentleman11
I looked at rclone once and didn’t find any information about encryption.
Thanks for mentioning it

~~~
geostyx
Crypt will encrypt all files before they leave your device, yes.

~~~
gravitas
To the non-rclone users, this refers to one of the targets of rclone; the
crypt backend wraps the real backend by chaining them together, e.g. localdata
<=> crypt <=> providerstorage (basically like a bi-directional filter).
[https://rclone.org/crypt/](https://rclone.org/crypt/)

Edit: I use rclone as the backend for duplicity, so you can also chain it
through another tool with different encryption and use rclone as just the
transfer engine, getting all the benefits of rclone's providers with the
benefits of duplicity's backup strategies.

------
sandworm101
I win. The 8TB drives I used to populate my NAS scored 0.10% better than the
0.81% average. That means my NAS is better than everyone else's.

This big takeaway from these numbers is how dramatically low they are in
comparison to drives 5/10/25 years ago. If you treat them reasonably, modern
drives are rock stable.

~~~
programmertote
Tangential question: What NAS device and set up do you use? I'm thinking of
buying one and would like to learn from a fellow (technically better
acquainted) HN user. Thanks in advance!

~~~
sandworm101
Synology DS1019+

It might not be the cheapest, but 4+1 drives is more efficient than 3+1.
Currently I have three 8TB drives: 1xWD Red and 2xIronwolfs (wolves?) with one
running as parity (SHR). No complaints. It does everything I need it to do.
The interface is simple and everything "just works" as anticipated. If/when I
run out of space I plan on adding another 2x8TB drives, probably more
ironwolves.

I wouldn't go bigger than 8TB as they already take DAYS to add to the array.

~~~
sumtechguy
I have that same box. Very nice box.

Migrated from a 1512. It took probably about 3-4 days to migrate everything.
The new box handled it easily. The old one was struggling a bit with rsync. As
the default is ssh with it. Once I had to downgraded the encryption then it
went a lot faster (from 30MB to 90MB which was close enough to the limit of
the cable to not worry).

------
astrophysician
I really enjoy reading the Backblaze analysis on this every year, it's such a
valuable and interesting data set. I do have one suggestion: it would be great
to go one step further and add confidence intervals for the AFR estimates.
E.g. if you see 0 drive failures, you don't really expect an AFR of 0 (that is
not the maximum likelihood estimate), and the range of AFR's you expect for
each drive decreases as a function of the number of drive days (e.g. if one
drive has 1 day of use, we know basically nothing about AFR so confidence
interval would be ~0-100% (not really, but still quite large), or it would be
smaller if you wanted to add a prior on AFR).

It would also be interesting to see the time-dependence (i.e. does AFR really
look U-shaped over the lifetime of a drive?). That would require a dataset
with every drive used, along with (1) number of active drive days, and (2) a
flag to indicate if the drive has failed and of course (3) which kind of drive
it is. Does Backblaze offer this level of granularity?

EDIT: They offer the raw data dumps!

[https://www.backblaze.com/b2/hard-drive-test-
data.html#downl...](https://www.backblaze.com/b2/hard-drive-test-
data.html#downloading-the-raw-hard-drive-test-data)

Backblaze, god bless you.

------
pwinnski
I never get tired of these reports, and if I ever again buy a drive with a
high failure rate for Backblaze, somebody slap me!

~~~
atYevP
Yev here -> Thanks! We love producing them - they are a lot of fun!

------
bronco21016
I've recently joined the WD shucking crowd so, unfortunately, there are no
stats for the drives I'm running. The cost/GB has just gotten too ridiculously
low on shucked drives and my array large enough that a failure or two isn't
the end of the world, oh and 3-2-1 backup rule.

I've always really enjoyed reading Backblaze's reports though and have made
past buying decisions based on their information.

I wonder if it might be possible for a community sourced version of these
reports? A small app that checks SMART data and sends it to a central
repository for displaying stats? Many from the self-hosted/homelab crowd are
running these shucked drives so there has to be a large pool of stats out
there if it can be gathered?

~~~
andy4blaze
Andy at Backblaze here. We've looked at this and the one "issue" is the
consistency of the environments from the community data. In our DCs, the
drives are kept a decent temperature, hardly ever moved, and our DC tech sing
to them every night (OK, just kidding about that last one). Community drives
will comes from all types of environments, from pristine to dust bunny hell.
Still, it might be interesting to compare the two data sets if a community
cohort could be collected.

~~~
touisteur
Be careful not to sing too loud _at_ them
[https://m.youtube.com/watch?v=tDacjrSCeq4](https://m.youtube.com/watch?v=tDacjrSCeq4)

Incidentally I've tracked down a HDD (spinning rust) performance (and later
breakage) problem to a very loud (and I mean awfully painful, intense
vibration on your eardrums) alarm test every wednesday...

Do you monitor the noise in your DCs? Maybe for high peaks?

------
jaclaz
This deserves to be cited:

HGST drives win the prize for predictability. Boring, yes, but a good kind of
boring.

~~~
codezero
Also worth calling out that the worst performer, Western Digital, acquired
HGST, now let's see if WD improves or HGST gets worse :)

Edit: See comment below, looks like this happened a while ago, so not likely a
real factor :)

~~~
dv_dt
WD acquired HGST in 2012, but was strictly independent until 2015.

[https://en.wikipedia.org/wiki/HGST](https://en.wikipedia.org/wiki/HGST)

~~~
codezero
Thanks a bunch for the correction, I guess the answer is that potentially the
acquisition has helped WD drives a little bit :)

~~~
dv_dt
FWIW I wonder if they're still run as something of a separate group, but given
that the article says the HGST brand was removed in 2018, who knows what speed
the long term merge might occur upon.

Explains why HGST brand drives are harder to come by I think.

~~~
codezero
Yeah, this wouldn’t surprise me, and it’s not super uncommon. Similar things
happened with Cisco and Meraki, though I’ve heard over time that’s started to
erode.

------
ramraj07
One of my main questions is how different these stats are for hard drives that
are not running all the time. Personal experience over a decade in an
underfunded lab was that if you took a drive offline and left it for couple
years, you had 10-40% chance of it failing.

~~~
leetcrew
I had a couple drives that no longer worked after leaving the computer dormant
for a few months. most of my drive failures have occurred immediately after
hardware upgrades though. seems like I lose at least one drive every time I
replace a mobo. I'm always surprised that breaks them, but they always
survived trips to/from college in the trunk of my car.

~~~
londons_explore
I'd give those drives a reformat... Some motherboards have firmware that does
odd things to drives, like hiding the first or last sectors to store its own
data or for various OEM recovery techniques.

That can make your data look corrupt. Yet reformatting (or just putting the
drive in with the old motherboard) will magically fix them.

------
java-man
Interesting how HGST has consistently low failure rates [0]. What makes these
drives so reliable? Japanese fixation on quality? Specifics of Hitachi design?

[0] [https://www.backblaze.com/blog/wp-
content/uploads/2020/08/Ch...](https://www.backblaze.com/blog/wp-
content/uploads/2020/08/Chart-Q2-2020-MFR-AFR-1024x679.jpg)

~~~
cehrlich
Interesting to note that this wasn't always the case - The Hitachi Deskstar
line used to have the nickname "Deathstar" due to their low reliability (this
was maybe 15-20 years ago)

~~~
zokula
The DeathStar name was never used when Hitachi made the drives. Deathstar was
used omwhen IBM used to make the drives

------
system2
Why wouldn't backblaze use only HGST then? What's the purpose of buying
seagates which was the most unreliable hard drives from 15-20 years ago and
their stats show higher failure rates. (Still I don't buy them because of the
bad taste left.)

~~~
atYevP
Yev from Backblaze here -> Some of the responses down below are correct - it
really does come down to price. We run a fairly lean ship and so the price per
gigabyte is weighed pretty heavily when we make our purchasing decisions. We
have a post where we go into the cost of hard drives over time, it's about 3
years old now, but still a good read -> [https://www.backblaze.com/blog/hard-
drive-cost-per-gigabyte/](https://www.backblaze.com/blog/hard-drive-cost-per-
gigabyte/).

~~~
toomuchtodo
Do you have an algorithm of sorts that takes into account price and historical
failure rate by drive model and gives you an easy "yea or nay" for purchase
decisions? Seems like you might be able to automate drive purchases with such
a system (similar to limit orders on financial securities).

~~~
atYevP
Yev here -> It's less of an algorithm and more of a purchasing department with
lots of spreadsheets that tie into the AFR data as a data point - but yes,
it's possible we could automate it, but there's some factors that our supply-
chain team would need to be involved in anyway (like vendor diversity, no
large batches of single drives etc...) that make outputting that algorithm a
bit harder.

------
ebg13
It's interesting that, despite Seagate being relatively bad compared to HGST,
Backblaze keeps installing more and more Seagate drives and not that many HGST
drives. I guess the analysis that's missing here is the cost per drive hour?

~~~
kube-system
It sounds like cost is the primary factor, but diversity can also help
mitigate the risk of unknown bugs.

Lets say you found that HPE SSDs were super reliable over your test period so
you decided to put everything on a bunch of new drives.

Then this hits, and 100% of your drives fail at the same time:
[https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-
fail-...](https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-
after-32768-hours)

While these stats are a great analysis of random failures -- that's only one
type of risk.

~~~
hinkley
I have seen homogeneity 'kill' twice in my career. The first time was
unprovisioned hardware, then second time we lost half of the hard drives
assigned for dev machines in the space of about 9 weeks. I had so many
processes in place by that point that it was a huge inconvenience (mostly due
to whole disk encryption) but zero data loss.

The worst thing you can do is to put all of your eggs into one basket. And
bulk ordering might get you a set of drives all from the same batch. Bad
batches will tend to all fail for a similar reason. Drives fail on a bell
curve, right? So the first drive may fail way before any others, but in a RAID
array, rebuilds are stressful. Eventually you will hit a statistical cluster.
Multiple drives failing close together. If that happens during a rebuild, you
will lose a RAID 5 array. If you are very lucky, your RAID 10 array loses two
drives in the same mirror. If you have two failures during a long rebuild,
even RAID 6 won't save you.

I just bought a Synology box for home. This is my third and probably final
RAID enclosure for personal use. I was having trouble finding Backblaze-tested
drive models to populate it, so I filled it with drives from a Drobo and kept
looking. Initially I had populated the Drobo with 4 drives I bought at once.
When one failed, I bought 2 HGST drives and replaced a pair. When the new
drives arrived, I started trying to cycle them through, and one of the drobo
drives failed. I'll give you two guesses which one.

There is, as far as I can tell, no prosumer multi-disk filesystem that uses
consistent hashing to stripe+mirror files across an arbitrary number of disks,
instead of the heavy linear algebra RAID5 relies on. It requires touching the
whole disk on every rebuild, and I believe that's why Object Storage is slowly
taking over from the top end. It's a simpler form of redundancy.

I hope that it's worked its way down to my price range by the time the
motherboard on the Synology burns out.

~~~
andy4blaze
Andy from Backblaze here. Nice thinking about the bulk ordering and
considerations for RAID. All things we have considered. We use our own Reed-
Solomon encoding with a 17/3 set-up across 20 drives across 20 different
systems, we call that a Tome. Then we have a specific protocol we follow as
drives fail in a Tome to protect the data at all costs. We have the luxury for
example to stop writing to a given Tome as we have plenty of others available.
This takes a lot of the stress off of the system. Your thoughts on bulk buys
and bad drive batches/models is solid. We test drives in small batches first,
and we follow drive failures so we don't get to the point of hitting the wall.
It would be great to mix and match drives, but you end up with a system that
maxes out at the least performant drive. So not optimal.

~~~
hinkley
I recall reading about your triple redundancy with R-S codes, but it's good to
restate it for each audience.

From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it
feels like repairing an array takes time proportional to the size of the
drive, not the size of the contents, and it feels like a waste. But it's
possible that my comfort levels for available disk space are a lot more
conservative than other people's, and so the difference is less pronounced in
a 'normal' storage situation.

For instance, an array that's at 80% capacity takes 25% longer to rebuild than
I wish it would, whereas an array that's at 66% capacity takes 50% longer.

~~~
TheDong
> it feels like repairing an array takes time proportional to the size of the
> drive, not the size of the contents

This is only true for drive-level raid rather than filesystem level raid, or a
non-raid solution like ceph's replication.

ZFS's filesystem raid can repair a raid in time proportional to the amount of
data stored in it.

mdadm and raid controllers aren't aware of which parts of the block device are
in use or not, and thus have to repair the whole drive.

It's exceedingly likely that backblaze's solution does not require repairing
entire block devices, but rather is likely to be closer to ceph, where only
the in-use portion of a failed drive must be considered / must find a new
home.

I think raid and distributed storage systems (like backblaze or ceph) are more
different than they are alike.

> From the behavior of my RAID (which also uses Reed Solomon, doesn't it?)

Maybe. mdadm raid5 doesn't, nor does mdadm raid1 or raid10. I think mdadm's
raid6 does.

------
gruez
Are there trends for AFR by drive age? ie. for a specific drive or
manufacturer, what is the failure rate for drives that have been in use _n_
years? It'd be interesting to see how the failure rate go up/down as they get
older.

~~~
chiph
Electronics typically follow a "bathtub" reliability curve. You'll have a
large number of early failures (unflatteringly called "infant mortality") then
the curve levels off for a long period of time, and then starts rising again
as the items start wearing out.

[https://en.wikipedia.org/wiki/Bathtub_curve](https://en.wikipedia.org/wiki/Bathtub_curve)

It'd be interesting to see if BackBlaze sees this in their drive populations.

------
ping_pong
Why did they stop buying WDC drives? Is there a known issue with them?

Also, why can't I find HGST drives for decent prices? On Amazon they are
either refurbished drives or crazy expensive prices for new ones?

~~~
Unklejoe
> they are either refurbished drives

Just a heads up: there's no such thing as a refurbished hard drive. It's not
economical. Instead, those drives are actually just used drives with the SMART
counters cleared.

I've been personally burned by this.

~~~
neilv
So they're just rolling back the odometer, and destroying records of accidents
(errors)?

~~~
kayson
Yes and no. It's more like they're "recertified". It's pretty easy and non-
invasive to replace the logic board, which can fix many problems that might
warrant an RMA. I've also gotten refurbs back from Seagate and WDC RMA's that
have new top covers on the drive (evidenced by a new, non-retail sticker, no
old sticker underneath it). Presumably they're doing some inspection of the
platters and heads before sending them out. These drives came with a 30day
warranty and cleared SMART data. But I would argue that its fair to reset the
SMART data after this kind of refurbishment/re-certification.

That being said, there are definitely some sketchy drive resellers on
marketplaces like Amazon who just clear the smart data on old drives and sell
them as refurbished, or even new.

~~~
jandrese
Recertified/Refurbished drives are never worth it IMHO.

~~~
Marsymars
I wouldn't pay for them, but I haven't found them especially unreliable when
receiving recertified drives for RMAs of my drives.

------
ggm
The volumes of units are high enough I think this is north of 'too small
sample size for statistical validity' So the question in my mind is, how
_significant_ is the variance in the failure rates?

Is this underlying manufacturing error tolerances, or is this
shipping/deployment effects, or is this .. Aliens?

------
chromedev
What is Wasabi doing different where they're able to outprice B2, and get
better throughput?

~~~
tln
I'm guessing you work at Wasabi and already know :)

Just looking at the pricing pages, pure storage is cheaper at B2, but the API
calls and egress are not free.

~~~
chromedev
No I don't. I'm genuinely curious how they might be saving money.

~~~
sliken
Wasabi has a minimum retention time of 3 months. So if you create a 1GB bucket
and delete it 5 minutes later you still pay for it for 3 months.

