
An Inside Look at the Backblaze Storage Pod Museum - sp8
https://www.backblaze.com/blog/backblaze-storage-pod-museum/
======
berbec
I'm always amazed by how open they are. Its great to see people like this
succeed.

My one BB question is how many data centers do they have? I know they have
great sharing tech to keep data online if a pod or two goes down, but how many
physical locations do they run?

~~~
brianwski
I work at Backblaze.

> how many physical locations to they run?

Two separate datacenters in the Sacramento (California) region, and one in
Phoenix (Arizona). We are trying to open a European (Netherlands) datacenter
this month or next month.

However, unless you take explicit action to copy your data to two datacenters,
any one file (or piece of file) is in exactly one datacenter. We believe the
data to be extremely DURABLE (survive), but if your strategy is to "host
content" in a highly available fashion where people will die if your content
is offline for an hour, we recommend you use two different providers with some
sort of fail over. Another alternative is to use a CDN (Content Delivery
Network) in conjunction with Backblaze. You can find out more info here:
[https://www.backblaze.com/b2/solutions/content-
delivery.html](https://www.backblaze.com/b2/solutions/content-delivery.html)

For backups, Backblaze advocates for a 3-2-1 backup strategy.
[https://www.backblaze.com/blog/the-3-2-1-backup-
strategy/](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/) This is
where you keep 3 copies of your data, 2 on site, and 1 in the cloud.

The exact system for how we achieve high durability is described in this blog
past: [https://www.backblaze.com/blog/vault-cloud-storage-
architect...](https://www.backblaze.com/blog/vault-cloud-storage-
architecture/) where any one file is striped across 20 separate computers in
20 separate locations in the one datacenter, where we can entirely lose any 3
computers and the data is completely fine and available.

We are COMPLETELY transparent on how we calculate the durability, we do the
math (including the assumptions) in this blog post:
[https://www.backblaze.com/blog/cloud-storage-
durability/](https://www.backblaze.com/blog/cloud-storage-durability/)

~~~
Johnny555
_We believe the data to be extremely DURABLE (survive), but if your strategy
is to "host content" in a highly available fashion where people will die if
your content is offline for an hour, we recommend you use two different
providers with some sort of fail over_

Do your durability metrics take datacenter failure or human error into
account?

Datacenter failures are rare, but they do happen and can cause data loss if
all of your data is in that data center.

Likewise, human error can cause cascading failures across a datacenter (or
beyond) if there are no firewalls between zones that prevent a single
person/command/software update from affecting all copies of the data.

~~~
manigandham
That depends on what you mean by failure. Are you talking about a data center
failing because every single machine inside blew up? Otherwise the common
failures like DC power outage or network drops are about availability rather
than durability. The data is stills safe on multiple drives.

~~~
Johnny555
I'm talking about the kind of failure that hit a Microsoft Azure data center:

[https://www.datacenterknowledge.com/microsoft/azure-
outage-p...](https://www.datacenterknowledge.com/microsoft/azure-outage-
proves-hard-way-availability-zones-are-good-idea)

 _“ but in this instance, temperatures increased so quickly in parts of the
data center that some hardware was damaged before it could shut down " ... "A
significant number of storage servers were damaged, as well as a small number
of network devices and power units.”_

~~~
manigandham
Yea if that damaged all the storage servers containing your data then there
would be data loss.

~~~
Johnny555
That's why I asked if that 99.999999999% durability number includes datacenter
loss. It's an unlikely failure mode but is it .00000000001% unlikely? I don't
know.

Given the fact that Azure lost a datacenter with this failure mode, I don't
think it's in the "likelihood of an asteroid destroying Earth within a million
years" ballpark.

Their durability page doesn't really clear it up, they say "Because at these
probability levels, it’s far more likely that ... Earthquakes / floods / pests
/ or other events" known as “Acts of God” destroy multiple data centers". But
from the post above: "any one file (or piece of file) is in exactly one
datacenter."

So it doesn't take multiple datacenter failures to lose data, just one unless
you explicitly copy your data to multiple datacenters.

------
skunkworker
It's crazy to see how fast storage prices have fallen. Just a couple days ago
I saw that you could get a WD White (Shucked easystore, pretty much a
relabeled WD Red) 10TB for just $169. 1.69 cents/GB. And you can get 8TBs for
$129 (1.61c/GB)

------
wiredfool
I hope it has a pile of the dead 3gb seagates. Perhaps in an interactive
exhibit where there are some implements of destruction.

~~~
atYevP
Yev from Backblaze -> We do have a drive crusher in the office, but mostly use
that for small externals. It's quite satisfying!

~~~
russh
Do you have to treat the output of the drive crusher as hazardous waste?

------
daveguy
I always enjoy reading about Backblaze updates. New Pod designs, drive
statistics, operations, etc... One thing I am particularly excited about is
the future Pod designs with 2.5 inch drives. There may be enough
miniaturization with magnetic drives to make this feasible, but I expect that
the real transition will come with a Pod full of SSD. Any idea when that might
happen? What is the expectation for that timeline? Do you have additional
products or price/performance improvements planned in that transition? The SSD
endurance experiment from 4 years ago indicates that, reliability wise, they
are more than ready. I guess the only limitation now is price and maybe
processing?

[https://techreport.com/review/27909/the-ssd-endurance-
experi...](https://techreport.com/review/27909/the-ssd-endurance-experiment-
theyre-all-dead)

------
chaostheory
The only thing missing from the museum are the people behind Storage Pod.
Other than that it's always really cool to see the evolution and history of
the product

------
DFXLuna
It's always fun to read stuff from the backblaze guys. The stuff they do is
just neat.

------
noir_lord
I love this kind of stuff.

I like the domain I program in but some of the problems in areas like this are
straight up nerd sniping[1].

[https://xkcd.com/356/](https://xkcd.com/356/)

