
Storage Pod 4.0: Direct Wire Drives – Faster, Simpler and Less Expensive - nuriaion
http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4/
======
andrewmunsell
Backblaze is great-- the product is fast and works well. But, there's one big
issue that prevents me from using it-- they delete your backups of your
external hard drives if they aren't plugged in for 30 days.

I have a drive that I may be away from (at least, my Macbook Air is) for weeks
at a time, and I don't want to continually lose version history or have to
reupload some of the larger files (home videos and my photos).

CrashPlan doesn't delete your backups after 30 days (they keep everything as
long as you're subscribed), but the upload speed is so horrendously slow I
couldn't upload the entire photo library plus videos (~300 GB or so) in the
month I was subscribed. I'm on a gigabit network over Ethernet, so it's not my
connection that's the issue (Backblaze, on the other hand, uploads this data
very quickly).

I realize that they're trying to reduce costs by deleting data that doesn't
seem to "be in use", but I'm sure a lot of people have "archival" data that
they don't look at often, but really value.

Instead, I'm currently using Arq 4 (which is fantastic, by the way) with
Amazon Glacier for the photos. It may cost something to retrieve them if my
drive dies or is physically destroyed, but I'm not planning for that to
happen. I have Arq set to use S3 for my other documents and such so I can
restore versions whenever I need to, so it's really the best of both worlds.

~~~
ghshephard
Agreed that this was one of the issues I had with Backblaze - I constantly
would lose all of my external drive backups when I went on a business trip,
come back, and all my backups would be gone and I would have to start over
again.

I finally ended up just purchasing multiple hard drives and using superduper
to clone/backup.

Crashplan wasn't an option - would take 60 days to do a 1 Terabyte backup, and
I don't leave my laptop connected to the externals that long.

I'd be willing to pay more money to backblaze to have them retain my external
hard drivers for a longer period of time.

------
dpe82
I love all the engineering Backblaze puts into this and their willingness to
share their experience.

I've noticed Supermicro offers a 45-drive 4u chassis* that costs more than the
Storage Pod's raw parts cost, but less than if you buy one preassembled by
45Drives. Does anyone have any experience with Supermicro's solution?

* Part: CSE-847E26-RJBOD1 ([http://www.supermicro.com/products/chassis/4U/847/SC847E26-R...](http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm))

~~~
budmang
Supermicro makes good systems that are very widely used and we love that
companies are continuing to work toward more dense and less costly storage
systems. When we started Backblaze in 2007, all the options were astoundingly
expensive.

One quick note on this particular Supermicro system - it's slightly
apples/oranges as the Backblaze Storage Pod is a complete server and this
Supermicro system is a JBOD, meaning it still needs to be plugged into a
server to work.

Gleb, Backblaze co-founder

~~~
nanch
At KeepVault we've been using Supermicro since day 1 (pre-2007) and they've
worked out very well. Supermicro is more expensive (on multiple metrics), but
you're also getting different features like entry-level cost and power
redundancy.

What happens when a Backblaze pod power-supply fails? I'd love to see a post
about that. :)

David, KeepVault CEO

~~~
derekp7
> What happens when a Backblaze pod power-supply fails?

From what I understand, the whole pod becomes unavailable. Which is why you
would use a front-end system to have redundancy across pods.

The way I would do it myself, is set up network connections between two pods,
and use DRBD, along with clustering software for the iSCSI or NAS (nfs/samba)
daemons.

------
kyrra
Side story: A company I worked for built a 60 drive unit similar to this. On
some of their early prototypes, they could hit power issues when all the
drives spun up at once (someone forgot to do their math). So all the
development on it had to be done with 3/4 the drives it actually supported.
Story #2: A 4U enclosure fully loaded with 60 drives goes beyond the shipping
weight limit for Fedex and UPS. You need to either freight ship it or not ship
it loaded with drives.

~~~
toomuchtodo
Did their SATA interface not support staggered spinups? Almost all do.

~~~
kyrra
I believe they did. It may have also done it when all drives were running at
max speed... I honestly forget at this point (it's been 7 years or so). It was
just the hardware guys screwing up on power distribution and calculated power
draw from the drives. Luckily they were prototypes.

------
theandrewbailey
Reading this, I realized more than ever that while SATA is a good hardware
standard, they messed up the naming. All throughout this post, mentions are
made to "SATA 3 throughput", which made me think they were referring to the
one with 300 MB/s transfers. Then further down, it was made clear that it
referred to the third SATA version, AKA SATA 6.0 Gbit/s.

At least we don't have to deal with master and slave jumpers anymore.

~~~
mikeash
Could be worse. USB speeds go, in order, Low Speed, Full Speed, High Speed(!),
SuperSpeed(??).

~~~
mkesper
Remember "Ultra High Frequency"? What will USB27.0 be called, "AbnormalSpeed"?
;)

~~~
radiowave
IIRC that one's going to be called "Ludicrous speed".

------
jobu
Looking at the parts list I see the largest cost is the 2x 40-port SATA cards
(aside from the drives). I wonder why they didn't decide to reduce the number
of drives to 40 so they could just use one SATA card and save $688?

~~~
atYevP
Yev from Backblaze here -> Density is key! Our biggest need is to fit as much
data as possible on each pod for as little price. The cost difference between
removing 1 SATA card or removing 5 drives is negligible, but the added drives
at another 20TB to the pod.

~~~
dpe82
Why not use a smaller card (or the motherboard's controller) for the last 5
drives?

~~~
atYevP
We're constantly tweaking the design, that might pop up in the future!

------
nulagrithom
"When a 5-drive backplane went down, we had 5 drives go offline. Now if a
connector goes down, only 1 drive will go offline."

Wouldn't a backplane be more analogous to a SATA card than a single connector?
In which case, 40 drives will go down on failure.

Neat stuff though. Would be fun to build one.

~~~
davis_m
The former model still had SATA cards that the backplanes connected to. I
think the biggest win for them is the fact that the backplanes were one of the
more error prone pieces in the box.

On a side note, each of the new SATA cards can support up to 40 drives, but
there are only 45 drives in the pod, so one failing would presumably take out
22-23 drives instead of 40. Still worse than losing 5 drives though.

~~~
justincormack
They still had 3 PCI cards before, as well as the multipliers, so card failure
is not much different. But port multipliers are all all the same bus, so more
error prone.

------
ksec
One of the thing i immediately noticed were the price /GB over the years. We
had literally no decrease since the flood in 2011. So three years later we are
still paying for about the same price per GB.

But that is finally going to change. As Google lower their price, i see this
as HAMR finally coming. With both Seagate and WD promised to get 20TB HDD by
2020, and with a 60TB version after. We have a clear roadmap of what is coming
along. Which means Google could now price their storage by it.

------
davis_m
I'm sure that the move to direct wire was done purely because of the former
backplanes being error prone, but the fact that they realize finding the parts
is difficult for others and mentioning it in the blog post shows that the guys
at Backblaze care about more than just making money.

~~~
atYevP
Yev from Backblaze -> We do! In fact we make $0 off of the pod design and why
we open sourced it. We want other folks to tinker, so we try to stay as open
as possible about the design and building of the pods!

~~~
derekp7
Question: Has any of your design improvements come from the community as a
result of being open? And do you end up getting better pricing from your
suppliers because there is a bigger market for this design?

~~~
atYevP
Yes actually! One of the first things that the community came back with after
Version 1 was that the hard drives vibrated a bit hard for most use-cases.
They recommended we use "hard drive vibration dampening sleeves" (rubber
bands) to make the drive fit a bit more snug. It worked like a charm! We don't
really get much better pricing, though sometimes we get a small discount
because we buy some pods/parts in bulk, though not in extreme quantities.

~~~
e12e
I really love these technical blog posts (and the designs, of course). Did you
ever calculate something like mean-failure rate for pods? Such data might be
helpful when evaluating if the architecture is viable for a certain use-case.
I'm thinking both disk-loss (which you've blogged about before), but also
aggregate failure rates (psu, controller cards etc)?

I'm thinking that for small-medium deployments, (with archiving in mind) three
pods would be a reasonable minimum, possibly starting them out sparse (single
radi6 to each pod, say)?

I'm aware there aren't any silver bullets :-)

~~~
atYevP
Not the failure rate of pods themselves, but we do keep track of the failed
drives within them. We did publish some blog posts about those stats, you can
find them on blog.backblaze.com! The pods as a whole are pretty stable, them
moving parts within them...not so much! Which is one of the reasons we went to
direct wired connections!

------
vicaya
Always love the practical hardware designs from these guys. I'm also
interested in some software stories: Linux kernel/distro used, issues
encountered/solved etc.

------
km3k
I love the look of everything Backblaze does, but I don't use them because I
need Linux support. I currently use Crashplan because of their Linux support,
but I find the service mediocre overall.

------
3327
This is great, I might actually build one. I love to see a company emerge that
doesn't take the status quo and hacks together a unique solution. This is
innovation that I love to see.

------
NKCSS
Small error in Appendix a; they list 2 zippy psu's (they only use one) and the
total also only count them once.

~~~
budmang
Fixed. Thanks!

------
wilhil
I'm so happy they give these plans for free. I have been trying to get
involved with Open Compute for some time and just get no where - with these
plans, anyone can get a head start.

Amazing and I hope they continue for a long time!

------
pronoiac
This is awesome! I like how they document everything. Tomorrow, I'm
interviewing at a different place, which is trying to deal with their storage
issues, and I was going to mention Storage Pod 3.0. So this is very timely for
me!

~~~
Xorlev
Just make sure they know they still need to manage data replication
themselves. You plan on entire machines going dark, not just drives.

------
kabdib
Not sure I'm okay with the single power supply. I guess if you can do
redundant storage across pods it'd be okay. But if that single PSU goes under,
you're in for a fun couple of hours.

~~~
mike-cardwell
This is better than their previous POD though, which had two PSUs, but no
redundancy. I.e, if a single PSU failed, the entire machine would go down as
it wouldn't have enough power to run all of the drives.

Obviously, for them it is cheaper to live without PSU redundancy because it is
a normal day for them to lose disks and PODs because they've got so many of
them, and they designed for this scenario.

------
DonGateley
What an amazing degree of disclosure. Tim Nufire is an engineer's engineer. He
gets me to thinking of how I might be able to justify the cost of one. :-)

