
Hard Drive Stats for Q3 2017 - LaSombra
https://www.backblaze.com/blog/hard-drive-failure-rates-q3-2017/
======
philfrasty
Three feedback points for B2. In case anyone from BB is reading:

\- (mentioned this before) invoices need improvement. No accountant will
accept this as of now. Please take a look at e.g. Digitalocean for how to do
this right. I mentioned this at least 5 times to support...but nobody seems to
care.

\- upload speed is really bad from Europe (might be purely due to distance).
Tried uploading one TB (@1.25MB/s), did around 25GB per day. Switched to a
Hetzner Storage box, doing consistently 75GB per day right now.

\- deleting your bucket (with data in it) requires to install a command-line
tool which is really inconvenient. Took me 30 minutes+ and still threw some
weird java.exception afterwards when clicking on „delete bucket“. Why not put
a simple „delete“ button in the web ui?

~~~
diggs
Deleting a bucket in an object store is surprisingly non-trivial. See Amazon’s
UI - it works fine for small object counts but is very buggy for larger
buckets.

~~~
atYevP
Yea, it's definitely a pain-point. Wish we had a more elegant solution, but
it's a tough problem to solve!

------
atYevP
Yev from Backblaze here -> at a conference today (JNUC2017 for those curious)
but I'll be checking in here to answer questions should any arise!

~~~
srcmap
Do you have any info on what temperature your drives are running on normally?

Love to see some performance stat to see if read /write IOP, MB/second, etc
change over time on per model base.

Other than complete failure, do you collect check the any other early warning
failure data such as ECC change over the life time of the drive? --- Would be
a fun big data exercise for an intern to work on.

~~~
atYevP
Hey there! We've written this about temperature before:
[https://www.backblaze.com/blog/hard-drive-temperature-
does-i...](https://www.backblaze.com/blog/hard-drive-temperature-does-it-
matter/). As for the performance other than total failure, we do keep an eye
on SMART stats -> [https://www.backblaze.com/blog/hard-drive-smart-
stats/](https://www.backblaze.com/blog/hard-drive-smart-stats/) and
[https://www.backblaze.com/blog/what-smart-stats-indicate-
har...](https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-
failures/) so we do monitor their general health as well.

~~~
srcmap
Nice info. Thanks.

Also what's the usage patterns for your HDDs? read/write/spin down/sleep time
in term of life time of the HDD.

I assume mostly write, some read. But also very low percentage is active?

~~~
atYevP
Mostly it's write - get fairly full - then sit until read :D

------
eljimmy
I love these reports! I'm always in the market for new storage solutions and
find these statistics very useful in determining which brands and drive sizes
provide the best reliability.

One statistic I'd really like to see included is the disk usage of each drive
as a percentage of their total capacity. Would there be any correlation
between that and the failure rate?

~~~
atYevP
If you look at the full data set it includes SMART stats which I _think_ has
percentage of space used and/or available. Not 100% sure about that.

~~~
eljimmy
Ahh yeah I was looking through the SMART stats but wasn't sure which item
denoted this, is it perhaps: "Minimum Spares Remaining - The Minimum Spares
Remaining attribute indicates the number of remaining spare blocks as a
percentage of the total number of spare blocks available."?

(Source:
[https://en.wikipedia.org/wiki/S.M.A.R.T.](https://en.wikipedia.org/wiki/S.M.A.R.T.))

edit: No that can't be it. Maybe this data isn't available, hmm...

------
nabla9
Maybe you could also add annualized failure rate per TB to the table. It seems
relevant when comparing different sized HD's.

When 12TB HDD fails, it's equal to four 3TB HDD's failing at once.

~~~
hinkley
Backblaze, from what I gather, mostly just leave dead drives in place for a
while.

They have to maintain more racks to do that (how much does it cost them to
have their drive arrays running st 90%?) but that drops the amortized cost of
replacing the drives.

But for me it might be cheaper to have one big drive that fails twice as
often, because the odds of a drive being dead on any given Tuesday and the
costs of addressing that might be low enough to offset the price differential.

------
throwawaysml
Yev, would you ever consider opening the software and hardware design of the
vault?

It seems that you could lead this and give the industry a standard way, with
vendors producing the fitting hardware, and allowing others to maintain their
own vaults. Maybe this is a business advantage you don't want to lose right
now. Then, I'm sorry to have asked :). If not then I think you might benefit
by lower prices if the hardware is readily available and not a Backblaze
custom production run.

There are many places who cannot for technical or won't for political reasons
offload their storage to S3 and friends and need it all locally most of the
time which would use a vault or two.

~~~
atYevP
The hardware design of the vault is open! The vaults are running Backblaze
v6.0 Storage Pods ([https://www.backblaze.com/blog/open-source-data-storage-
serv...](https://www.backblaze.com/blog/open-source-data-storage-server/)) and
the nice thing is, you can use whatever software you want to run them (we
likely won't release the software, but we have talked about parts of it like
Reed-Solomon Erasure Coding [https://www.backblaze.com/blog/reed-
solomon/](https://www.backblaze.com/blog/reed-solomon/)).

As for getting the servers, there's an entire company that popped up from one
of our early storage pod producers, Protocase, you can buy servers (very
reasonably priced btw) based on our designs here ->
[http://www.45drives.com/](http://www.45drives.com/).

------
IronBacon
I have a somewhat off-topic question: are the new 10/12 TB HDD filled with
helium (He)? Last time I've checked on WesternDigital and Seagate sites I
couldn't find any mention of it.

~~~
atYevP
Yo! I believe the 10TB ST10000NM0086 ones are helium!

~~~
dghughes
That's so the pods can float right? ;)

~~~
atYevP
If the pods are floatin' the sys admin's dopin'... (or some other copin'
mechanism) :P

~~~
zaroth
Aside: HN needs a way to indicate hard line break which doesn't cause 1.5
spacing, aside from the code indent with its too-narrow scroll window.

------
paulmd
I see Seagate is continuing their track record of Being Shit, as usual. Yet
again they're the only brand where a couple models are exceeding the normal
failure rate by a factor of 50-100x.

~~~
Joeri
If they’re always terrible why does backblaze keep buying them?

~~~
paulmd
To add to the other comment, it's not just price but also volume. It's easy
enough for me as an individual to buy 10 Toshiba HDDs... but Backblaze would
like to buy them in quantities of 10k or more (if at all possible - obviously
during the Thailand Drive Crisis they had to deal in small quantities).

It's a lot easier to get Seagates in large volumes than some of the other
brands - Toshiba specifically is problematic, iirc HGST maybe also.

Also, I didn't say Seagates were always shit. But some of their models are,
and they are the only brand with these issues, and it's been a repeated
problem for them over the last 10 years.

------
jumpkickhit
Those 10gb Seagates have to be pretty new still. I'm curious about them over
time. It would be nice if that's one like their old pre 2006 hdds that hardly
ever failed.

~~~
PascLeRasc
Anecdata, but my work's just passed the 1 year anniversary of running 12 of
those and they're all still in perfect health. I'm very impressed by Seagate's
enterprise drives.

------
CaliforniaKarl
Upvoted, but I'd like to suggest a title change, since this is just the Q3
report. The next report will be covering all of 2017.

>Our next drive stats post will be in January, when we’ll review the data for
Q4 and all of 2017, and we’ll update our lifetime stats for all of the drives
we have ever used. In addition, we’ll get our first real look at the 12 TB
drives.

~~~
dang
Changed, thanks!

