
Backblaze Drive Stats: 2018 Q3 Hard Drive Failure Rates - LaSombra
https://www.backblaze.com/blog/2018-hard-drive-failire-rates/
======
jewel
A total of 704 petabytes of raw space, which I think is 600 petabytes of
usable space.

Looks like backblaze will cross the exabyte mark very soon. They crossed 500
PB in February [1].

Yev, in case you're around, I was curious what your plan was for a potential
future shortage of supply of hard drives due to another disaster, political
problem, etc. Perhaps just grow far enough ahead of your needs to have a
buffer? I imagine you're now at a scale where you can't farm hard drives from
costco anymore.

Also what happens to all of those functional, but non-profitable 4TB drives?

[1]: [https://www.backblaze.com/blog/wp-
content/uploads/2018/01/50...](https://www.backblaze.com/blog/wp-
content/uploads/2018/01/500pb_growth.jpg)

~~~
atYevP
Yev from Backblaze here -> of course I'm around :D Good questions! We recycle
the drives that get removed from service. As for future shortages, that's one
of the reasons we took a small round back in 2012
([https://www.backblaze.com/blog/backblaze-raises-5-million-
wh...](https://www.backblaze.com/blog/backblaze-raises-5-million-why-we-took-
funding-after-5-years-of-bootstrapping/)). The majority of it was to spend on
marketing experiments, but a little portion was to be used for buffer if an
event like that ever happened again and hard drive prices spiked 300%. We're a
lot larger now and do a pretty good job with forecasting and having rainy day
funds, but we've also gotten to the point where we are now a bit less price
sensitive, so we'd be able to handle the fluctuation. That said, if everything
goes horribly
wrong...[https://www.backblaze.com/blog/backblaze_drive_farming/](https://www.backblaze.com/blog/backblaze_drive_farming/)

~~~
wlesieutre
You might not be the right person to do anything about this, but I have a
bookmarklet for hiding position:fixed elements when I'm trying to read
something, and every time I scroll your blog it recreates the obnoxious share
bar. Whyyyyyyy?

EDIT: PS- fun story! I enjoyed reading it except for all the social media
buttons.

~~~
atYevP
Hahahaha! Social sharing is great! Do not fear the share bar! Seriously
though, sorry about that - not sure if we can do anything on our end as that's
the way that widget works :( We might be able to add a "print" button so you
can just print the post out, would that help?

~~~
wlesieutre
It's not that I never share things, I'd just rather ctrl-C/ctrl-V than waste a
bunch of screen space on every single webpage.

Looks like Firefox's reader view fixes it, I'll just need to get in the habit
of using that instead of the Kill Sticky Headers bookmarklet as things catch
on and circumvent it... [https://alisdair.mcdiarmid.org/kill-sticky-
headers/](https://alisdair.mcdiarmid.org/kill-sticky-headers/)

I can only assume the thought process of whoever writes these social bars is
"People are hiding our share buttons instead of clicking them! MAYBE IF WE
SHOVE IT IN THEIR FACES REPEATEDLY THEY'LL LIKE THAT BETTER AND WANT TO
SHARE!"

Gruber sums up my feelings nicely:
[https://daringfireball.net/2017/06/medium_dickbars](https://daringfireball.net/2017/06/medium_dickbars)

~~~
bubblethink
>Gruber sums up my feelings nicely:
[https://daringfireball.net/2017/06/medium_dickbars](https://daringfireball.net/2017/06/medium_dickbars)

Ironically, that site is perhaps more unreadable than a typical medium page.
It uses some hardcoded mobile format (?) that shows tiny text in the middle
20% of my 16:9 screen.

~~~
wlesieutre
And you can't just double tap the text column to fit it to screen? That's odd,
since it doesn't appear to set a maximum-scale or user-scalable=no on the
viewport:

    
    
        <meta name="viewport" content="width=500, minimum-scale=0.45" />
    

#worksonmymachine

------
arminiusreturns
As a sysadmin I can't say how much I appreciate BB publishing these each year.
it helps me make informed decisions on drive selection backed by data.

~~~
atYevP
Yev from Backblaze -> Glad you like it! *Edit -> I was just at SpiceWorld in
Austin and a lot of the sys admins who were there enjoyed it quiet a bit! I'm
headed to JNUC in Minneapolis next week and last year we talked a lot about
hard drive stats as well!

------
xmichael999
I always enjoy this site for anyone in the market.
[https://diskprices.com/?locale=us](https://diskprices.com/?locale=us)

FYI I have nothing to do with the site, just a user.

~~~
jaclaz
Sure, but, at least for my local, ALL links are to Amazon.

------
Dennip
Do BB publish pricing of what they paid for drives? Sort of a TB/$/Failure
rate. Those 12TB HGST drives must cost a ton

~~~
brianwski
Disclaimer: I work at Backblaze.

> Do BB publish pricing of what they paid for drives? ... a TB/$/Failure rate

Thank you, that is EXACTLY the correct way to look at the failure statistics!
So many people seem to sort the list by failure rate and think no matter what
the cost, the lowest failure rate wins. For Backblaze, we just feed it back
into the cost calculation. For example:

If a drive fails 1% more often but is 2% cheaper in total cost of ownership,
we buy that drive. Now, total cost of ownership includes the physical space
rental so more dense drives can be more expensive per TByte in raw drive cost
because we can make some of that back up in physical space rental. Also, most
drives seem to take about the same amount of electricity unrelated to how many
TBytes are contained inside, so double the drive density is like saying it
takes half as much electricity over its 4 - 5 year lifespan. Electricity is
one of our largest datacenter costs, so we keep an eye on that also.

But to answer your very first question, unfortunately we cannot release the
price we paid for the drives due to the vendors requesting we don't disclose
prices. But don't think we have some magically huge discount or anything. Most
of the time we are literally paying about retail, maybe a 3% - 5% discount for
buying in bulk orders of over 10,000 drives. But for reasons nobody at
Backblaze can figure out, sometimes a bunch of new drives appears randomly at
a really good discount price, then returns to the original price the next
month. It might be some attempt along the supply chain to boost monthly or
quarterly top line at the expense of profits, I don't know.

~~~
brokenmachine
When I'm just buying drives for my home machines, I'd pay much more for a
lower failure rate because a dead drive means a lot of time and massive
inconvenience plus some research and a trip to the computer store to buy a
replacement.

I'd honestly pay double if I could be guaranteed to avoid all that.

~~~
ADefenestrator
There's some other downsides of course, but RAID1 basically gets you exactly
that. Double price for much lower failure rate of the storage volume. Go with
different manufacturers for the two drives to further reduce the failure rate
at the probable cost of a little performance and space.

------
chrisper
Any plans on the European Datacenter?

~~~
atYevP
Yev from Backblaze here -> yes but no ETAs at the moment!

~~~
mjevans
That would be spectacular; I know this is obvious, but please do make sure
that it actually can operate independently of the other locations and that end
users have an easy way of controlling which region their data sits in.

There are legal reasons some companies might need to keep their data within
the EU, and for others having multi-region backups are a checkbox to tick off
on disaster preparations. Both are things the marketing department should use.

------
swang
What does Backblaze do with all its older hard drives? Resell? Recycle?

~~~
brianwski
Disclaimer: I work at Backblaze.

> What does Backblaze do with all its older hard drives? Resell? Recycle?

If the drive is still working (like you can read and write to it) when we
cycle it out, we securely wipe it and then sometimes resell them in "bulk" to
places like "Weird Stuff Warehouse" in the San Francisco area, or other
places.

If the drive is not working, we physically destroy the hard drives (special
equipment) and dispose/recycle the electronic waste of the carcass.

~~~
corobo
> physically destroy the hard drives (special equipment)

Is that as fun a day as I'm picturing?

~~~
dsfyu404ed
When you've got hundreds or thousands of drives to destroy it's not worth the
man hours or consumables to cut them or drill holes in them. Even shooting
them gets old quickly.

When I worked for a defense contractor we started one of our daily infosec
briefings with a video of one of our shipyards destroying a few hundred hard
drives. They put them all in a shipyard size press brake. The end result was
very long line of C shaped hard drives.

Anyone who deals with enough drives and doesn't have other business (like a
shipyard) that they can borrow suitable equipment from will likely wind up
buying an industrial shredder.[1]

[1]
[https://www.youtube.com/watch?v=sQYPCPB1g3o](https://www.youtube.com/watch?v=sQYPCPB1g3o)

------
vxNsr
What do you guys do with decommissioned but functional hard-drives?

~~~
rtkwe
Like google probably wipe and maybe shred. I thought they were all shredded
but it turns out they do sell decommissioned drives if they can be verified as
100% wiped.

Google actually recently upgraded to using a robotic arm to manage the wiping
and shredding of drives. [0]

[0] [https://www.datacenterknowledge.com/google-
alphabet/robots-n...](https://www.datacenterknowledge.com/google-
alphabet/robots-now-annihilate-hard-drives-google-data-centers)

~~~
WaxProlix
Little confusing, are you referring to shred the tool or some physical
destruction process?

[https://linux.die.net/man/1/shred](https://linux.die.net/man/1/shred)

~~~
sithadmin
Physical destruction. Destruction of decommissioned drives by shredding,
crushing, or passing a bolt through the drive is a standard practice for
security-sensitive operators. Some IaaS operators I've worked with extend this
to every component of systems that could ever hold customer or proprietary
data in memory.

------
antman
Something seems to go wrong with WDC 6TB.

~~~
breakingcups
It's a relatively small sample though. 5 failed drives.

------
hanyoloren
cool

