
Backblaze's Hard Drive Stats for Q1 2018 - LaSombra
https://www.backblaze.com/blog/hard-drive-stats-for-q1-2018/
======
justinsaccount
Kinda interesting.. if you multiply Drive Size by Drive Count and sum the
result, you get 625 PB. Wonder what the expected time until they have a full
Exabyte of raw storage is

~~~
atYevP
Yev from Backblaze here -> Probably year end :P

------
foldor
Does Backblaze support Linux yet? I have store almost all of my data on a NAS,
and it would be great to have that automatically backed up online, but the
last time I looked into it they only supported Mac/Windows. I assume the
reason for this was to keep out users with large NAS capacity and allow them
to keep advertising unlimited storage. I would be ok with having Linux
computers capped, I'm not storing more than about 2TB of data.

~~~
qeternity
As much as I love B2, their SDK support is pretty terrible. There are a
handful of half-baked and abandoned libraries. We've been using an internal
python library which we're starting to open source and plan on maintaining.
We've also forked wal-e to support this library and B2 endpoint (currently
have basic upload functionality working). It would not take much effort from
Backblaze to throw some resources behind these efforts, or hell, to host their
own S3 proxy (I'll never understand the design choices behind their API).

~~~
brianwski
Disclaimer: I work at Backblaze.

> Backblaze's APIs are not S3 compatible. .... I'll never understand the
> design choices behind their API

It is all clear if you understand our APIs are cheaper to implement than
Amazon S3's APIs. I freely admit the APIs are LITTLE BIT more complicated than
S3 (which is not a good thing). Let me start from the beginning and explain
why:

We originally never planned to open the APIs up for general use. We created
the Backblaze Personal Backup client and we owned both the client and the
server, so we were originally willing to do a tiny bit more programming to get
rid of some expensive equipment and choke points in the datacenter.

In the S3's APIs, you simply "upload" and you are done. This means there is a
very high availability choke point on upload. With Backblaze client (and later
"B2" APIs), there are no choke points. The Backblaze client contacts (HTTPS) a
central dispatch server and asks the dispatch server for a pod (computer) that
contains some spare disk space. Then something very very important occurs that
does not occur with Amazon S3 -> the client entirely stops talking with the
central dispatch server. And then the client contacts the pod DIRECTLY to
perform the uploads.

Now the "contract" with the storage pods is that if that one storage pod
crashes, or fills up, or decides not to talk with the client anymore, the
client is absolutely responsible to go back to the "dispatch server" and
request a new pod somewhere else in the Backblaze datacenter to upload data
into.

By requiring these one or two extra steps, Backblaze does not have to buy any
expensive load balancing hardware, and we only need 10 Gbit network cards,
there are no central choke points requiring multi-Terabit networking speeds
like Amazon. Instead we have several thousand individual 10 Gbit network cards
which cost us almost nothing because they come built into the Intel
motherboards nowadays. The switches cost a few bucks per port.

I assume that Amazon S3 accepts the data, then has to "move it" to the final
destination with a network copy. In Backblaze, the data literally lands in the
final destination coming straight from the client. Fewer copies means lower
cost and higher performance.

Backblaze purchases no load balancers, except for some "tricks" to make the
"dispatch servers" highly redundant. But the dispatch servers have only TINY
trickles of info coming and going, and the clients only have to contact the
dispatch server ONCE every few days (then upload endlessly to the final
destination pods), so even the dispatch servers can be "only" 10 Gbit.

> Backblaze is incompatible with Amazon S3

TL;DR - by requiring clients to do a small amount of extra programming and
making programmers understand the "contract" (which is the client must retry
some things and handle a few more errors), Backblaze cuts out a lot of cost
out of our datacenter while INCREASING scalability over S3. We pass the money
savings along to our customers. We throw in the increase scalability over S3
for free. :-)

------
Damogran6
Interesting thoughts on SSDs. I was under the impression that it wasn't so
much the speed improvements they provide as the _power_savings_ that made SSDs
useful in a Data Center environment. It was a single piece of anecdotal
evidence, but the payoff due to power savings was roughly 8 months.

~~~
yellowapple
It depends on the use case. IO-heavy workloads (like databases) benefit from
the better performance (relative to traditional hard disks), though ramdisks
would give even more performance gains (so long as your data - or the most
actively used subset thereof - fits in RAM).

Another factor is the form factor (pun intended; I'll be here all week). SSDs
can get a lot smaller than traditional hard drives, which opens up the
floodgates for tiny servers (blades, single-board machines, etc.) as well as
for increasingly-dense solid-state SANs.

But yes, power savings are yet another huge benefit - and one which is still
such even as a side effect of going with SSDs for a different reason (like
physical space or needing high-speed disk access).

------
Flockster
In the section "Lifetime Hard Drive Reliability Statistics", shouldn't the
value 0.10% in the high confidence column be greater than the low value?

~~~
davrosthedalek
Yes, low/high should bracket the annualized failure rate. It's probably 1.1%

~~~
atYevP
Absolutely right, we've fixed the error! Whoops!

------
samfisher83
HGST seems to be doing pretty well. Maybe Western Digital used some of their
manufacturing techniques after they bought them?

------
kyriakos
The problem with the statistics provided is that they don't account for how
long the drives have been running. Eventually all hard drives will fail but
what matters is how long it will take until they do. From the charts I can't
understand which is the most reliable over time. Or am I just misinterpreting
the data?

~~~
24gttghh
It also doesn't seem to take into consideration the use-case for the drives.
Are they in write-heavy or read-heavy roles? Is it balanced between? Are there
regular, punctuated periods of heavy load? Is every drive in the same style of
SAN/RAID across the enterprise? (doubtful on that last one)

Maybe it doesn't matter due to the sheer number of drives being reported on
here?

~~~
Twirrim
Backblaze uses Reed-Solomon erasure encoding.
[https://www.backblaze.com/blog/reed-
solomon/](https://www.backblaze.com/blog/reed-solomon/)

Loosely speaking files are encoded in to to chunks and scattered across
multiple disks (actual stored data is more than the size of the original
object, but less than full duplication, triplication or other redundant ways
to store it).

That blog suggests Backblaze use a Reed-Solomon ratio of 17:20 ratio, i.e.
every object is encoded in to 20 chunks, for which you only need to retrieve
17 different chunks to be able to decode and get the full object.

If they're putting only one chunk per drive, that gives any object stored
resilience from up to 3 disks failing, at a storage cost of only 1.17x
original object size (20/17 ~= 1.176). They'll likely have repair processes to
proactively handle any loss of shards from a disk failure, too.

Reed-Solomon is giving them something akin to triplication, while using
significantly less disk space than duplicating.

Nothing comes for free, though. The additional cost here is the overhead of
encoding and decoding the object, and on top of that the co-ordination etc
required from requesting chunks from multiple locations. That adds some
unavoidable latency, but that article suggest Backblaze find the JVM to be
extremely fast (as fast as a pure C implementation of Reed-Solomon).

~~~
vxNsr
This is more interesting than the actual post, I didn't even know about this
type of encoding, thanks for giving me some weekend reading :)

~~~
atYevP
Yev from Backblaze here -> if you don't want to read it, we made this handy
video ->
[https://www.youtube.com/watch?v=jgO09opx56o](https://www.youtube.com/watch?v=jgO09opx56o)

------
jsgo
This is kind of depressing. Per the stats here if I'm reading it correctly, it
appears WD Red disks have pretty high failure rates relative to competitors
and that is what I've been buying to put in my NAS.

Why is there such a general WD > Seagate slant if the stats don't seem to back
it up?

~~~
sigstoat
a 2015 survival analysis of the data had WD looking better.
[http://blog.applied.ai/survival-analysis-
part3/](http://blog.applied.ai/survival-analysis-part3/)

it'd be nice if folks would repeat the more in-depth analyses more often, as
they can produce better answers to the questions folks really want.

------
leowinterde
The upcoming blog post about helium-filled drives sounds very interesting.

~~~
atYevP
Yev from Backblaze here -> You won't have to wait long ;)

------
agumonkey
Interesting improvements from Seagate.

