
Backblaze Storage Pod 5.0 - ingve
https://www.backblaze.com/blog/cloud-storage-hardware/
======
howeyc
Interesting, with this information and some idea of what they pay for
electricity +support staff/drive replacements/maintenance you could probably
figure out their break-even mark.

Their cost for the pod (which they say includes labor) comes out to 0.044/GB.
The cost of redundancy is 3/20 drives, which would place the cost to 0.0517/GB
of data. They are planning to charge 0.005/GB on the B2 service this is made
for. That's 10-11 months to break even on the initial cost. Add electricity +
maintenance costs (of which I have no idea what it is) and you get the break-
even number. My guess is that it'd be 1 year. So however long each pod lasts
past 1 year is gravy (aside from the electricity + maintenance costs).

Amazon S3 must be making a killing. As a customer you must be paying for the
cost of the "pod" every 2 months.

Please excuse me while I buy stock in Amazon/Box/Dropbox/Backblaze/etc....

\----

As a aside, as a personal user if you store over 1TB you are getting storage
at a cheaper cost than B2 users. If under 1 TB you'd be better of using B2.

Seems to me that the average storage per user must be well under 1 TB on their
unlimited plans.

~~~
budmang
Excellent analysis. There are some other costs involved (bandwidth,
switches/cabinets, non-pod servers, etc.), but pods & drives dominate the
costs, so your math is good.

You're also right that 1 TB is the approximate breakeven between Backblaze
per-GB cloud storage & unlimited online backup by cost and most people are
below that. However, the main reason for people to use the backup service is
that we take care of all the backup functions (encryption, dedup, compression,
restores, etc.)

> Please excuse me while I buy stock in Amazon/Box/Dropbox/Backblaze/etc....

Alas, Backblaze stock isn't yet available for sale ;-)

BTW, thanks for being a customer!

Gleb

~~~
atYevP
> Alas, Backblaze stock isn't yet available for sale ;-)

YET...or is it? /runs over to SecondMarket.

------
wyldfire
It's astonishing that Backblaze would release its drawings as open source,
good for you!

But there's no explicit license declaration anywhere in the drawings zipfile
and in fact some of the drawings claim "PROPRIETARY AND CONFIDENTIAL THE
INFORMATION CONTAINED IN THIS DRAWING IS THE SOLE PROPERTY OF BACKBLAZE
MANUFACTURING. ANY REPRODUCTION IN PART OR AS A WHOLE WITHOUT THE WRITTEN
PERMISSION OF BACKBLAZE MANUFACTURING IS PROHIBITED".

Maybe it's just boilerplate? Or maybe you really don't want to face
competition in your specific industry. Regardless, an official open source
license (even with commercial restrictions) would be much clearer than this.

~~~
atYevP
Yev from Backblaze here -> That is boilerplate for the most part, but it's
open source, have at it!

~~~
wyldfire
Ok I will email my attorney that "Some guy named atYevP on HN told me it was
okay." ;)

~~~
greglindahl
Having sold a company to IBM, I can attest that this issue would be a Big
Problem. {Edit: I mean for any company that used BlackBlaze diagrams labeled
"proprietary."}

~~~
wyldfire
A big problem for BB if they were to have open source-licensed tech and got
acquired by a megacorp, or a big problem for a "kinda-sorta-licensee" of their
tech once acquired?

~~~
moey
It was a joke, emphasis on IBM 'Big" stuff...

~~~
greglindahl
That was the pun, yes, but my comment was about a real issue.

------
mrmondo
Great work Backblaze! I really appreciate the open, knowledge sharing nature
of your hardware designs. This is one of the main reasons I am a Backblaze
customer, I must say however that the lack of open source and headless Linux
client software is seriously pushing me away, it's something I think many
people have been waiting for years for, especially given the product licensing
model is designed as such to backup a single machine only.

~~~
atYevP
Yev from Backblaze -> We're working on it, in a fashion. With Backblaze B2
you'll be able to push from a Linux machine using CLIs/APIs and get those
computers backed up as well! Slightly different, but it's the best way for us
to help Linux users!

~~~
sharmajai
It is probably unfair to ask, but is there a way to get an early invite for B2
(I already signed up but can't contain my excitement; email: HN username at
gmail)?. Also, have there been any comparisons of B2 done with Amazon Cloud
Drive's unlimited everything plan for $59 a year (with break even when
compared to B2 at around a TB)? They do provide an API, although I have yet to
try it out.

------
jamwt
Worth pointing out (for those of you not in the storage game) that these
appliances, while cool, are pretty far from state of the art. We're (Dropbox)
packing up to an order of magnitude more storage into 4U. And costs aren't
close, let alone the addressable throughput given that you guys just went to
10G.

Serious kudos though are due for the Backblaze blog. You're all at least
talking about what you're up to, which is awesome both for education around
building large storage systems and establishing transparency with your
customer base.

Unfortunately, a lot of the bigger clusters' teams won't/can't talk about what
exactly they're doing and how they're doing it for trade secret reasons. If
you want to learn about it, you have to come work for one of us. :-)

Also, the orangered hardware looks awesome.

~~~
codemac
What order of magnitude? There are 60 drive chassis in 4U you can get[0][1],
but even a 4TB -> 6TB drive + 15 more drives isn't exactly an order of
magnitude.

And they have 2 10GiB ports, though I'm sure you can work with them and get
more..

It's disingenuous to call this not "state of the art" when it's really quite
close, and obviously is meeting a backblaze design goal.

[0]:
[http://www.newisys.com/Products/4600.shtml](http://www.newisys.com/Products/4600.shtml)
[1]:
[http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H](http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H)

~~~
chx
A 96 bay 4U chassis is readily available
[http://www.raidinc.com/products/object-
storage/ability-4u-96...](http://www.raidinc.com/products/object-
storage/ability-4u-96-bay) but that's still not a magnitude.

~~~
atYevP
Heh, any place where I have to request a quote before they show me a price
makes me nervous :D

~~~
epistasis
It should if you're thinking of buying one-offs. But at bigger scales like
yours, you can either invest in crew to design chassis, or in purchasing
people to deal with other sales people.

------
masonhipp
I find these breakdowns fascinating. As an enthusiast PC builder I
occasionally consider running my own data-systems or servers... In the end
though I'm glad I have guys like you putting these together: clearly an
enormous amount of thought going into it. Thanks for the write-up.

~~~
atYevP
Yev from Backblaze here -> You're welcome! We're always pretty stoked to do
these, even though it does indeed take a lot of man-power to put together. The
nice thing is, it's actually helpful! A lot of organizations, companies, and
research labs use them, and that makes us feel warm and fuzzy.

~~~
toomuchtodo
I use a Backblaze box as an Archive Team staging environment! Thank you!

~~~
budmang
Very cool. I remember an article I read about a veteran music journalist who
applied for a job and was told he had no experience. Turned out every magazine
he worked for over 10+ years had shut down and there was not a single online
reference of his articles.

Love that you're fighting the good fight (along with the Internet Archive) to
keep a history of the Internet.

~~~
greedo
From 2000 through 2006 I freelanced for a middle tier publisher of computer
magazines. Wrote 4-5 articles a month, probably 1/2 million words. The
publisher had a paywall for their online versions of the print magazines, but
really didn't put much effort into promoting it since print had been so
lucrative. Over the last three years they shuttered the majority of the print
magazines (and their corresponding online editions), and now it's all gone.
The wayback machine/Internet Archive only has a limited number of pages
available. Sad to realize that a lot of the work we do (whether it's writing
or creating systems) is ephemeral...

------
budmang
I'm particularly excited about this version because it is the underpinning of
B2 (which required higher performance), but we were still able to keep the
costs low.

~~~
atYevP
^ -> Gleb is Backblaze's CEO. (I too, work for Backblaze).

~~~
merb
the bad thing about your good mission is that big guys starting to buy cheap
hdds which means that prices will go up :(

~~~
atYevP
Prices for the HDDs themselves? Doubtful, but maybe. The nice thing about
consumer-grade HDDs is that consumers are not very price elastic, meaning if
consumer HDD prices go up too much, consumers will just stop buying them (we
saw this happen after the Thailand Drive Crisis). Hopefully what it'll mean is
that drive manufacturers will instead start making lower-cost "professional"
drives for this exact purpose. Cheaper than "Enterprise Drives" but slightly
"better" to justify a slightly higher price than their consumer counter-parts.
Hopefully...

~~~
voltagex_
I still haven't been able to work out if prices ever went back to pre-Thailand
prices. It pretty much coincided with the beginning of the Aussie dollar
becoming worthless so that's skewed things a bit.

~~~
atYevP
Well, at least in the States it's gotten pretty close to those levels if not
gone down to slightly below them. Not sure where that relates to the Aussie
dollar though :-(

------
KNoureen
The main problem I have with Backblaze and AWS is that there is no budget
limit on transfers.

I would like to tell the provider; limit downloads to $500, then reply 509 to
further requests.

~~~
atYevP
Backblaze B2 will have that, we'll have spending thresholds in place.

------
Qiasfah
I notice that they're using a 500GB 5400RPM boot drive.

Is there a need for that large of a boot drive? I would have expected that an
SSD boot drive would be preferable.

~~~
atYevP
Yev from Backblaze -> We don't need the boot drive to be that "large", but
it's more of a pricing thing. An SSD, even at 256GB would be more expensive
than those 5400RPM drives, so we just go with what's cheap :)

~~~
greglindahl
Reliability has a cost, too. You can get a 60GB SSD for cheap (Amazon suggests
it's about the same price as the slow 500GB HDD), and the SSD will fail at a
much lower rate than that HDD.

~~~
DanWaterworth
When you have another 60 HDDs in the same case, I guess you think about
reliability differently.

~~~
greglindahl
The 60 HDDs aren't single points of failure. {Edit: I mean for the server, not
for the whole system.}

And a raid1 pair of HDDs for the system disk is more expensive than a small
SSD, more fussy, and the SSD is still less likely to totally fail.

~~~
DanWaterworth
It doesn't sound like the boot drive is a single point of failure since data
is stored Reed-Solomon coded in chunks across many pods. If one data drive
fails, the whole pod has to go down for maintenance for it to be replaced. The
only difference is that you get to choose when to take a pod down for
maintenance to replace a data drive.

~~~
greglindahl
That's the situation I was talking about, yes. It's much cheaper to go into
the datacenter once a month to replace failed data disks than to have to go in
to promptly replace any system disk HDD that fails, in order to not have 60
idle disks.

~~~
DanWaterworth
Sure, that would make sense if they only had personnel in the datacenter once
a month, but: "we replace about 10 drives every day" [1].

[1] [https://www.backblaze.com/blog/vault-cloud-storage-
architect...](https://www.backblaze.com/blog/vault-cloud-storage-
architecture/)

~~~
DanWaterworth
To clarify, I'm not saying that they wouldn't be better off with an SSD boot
drive. I'm arguing that having an HDD boot drive, given their setup, isn't
awful.

------
eiopa
I really love these, and I desperately wanted to be a Backblaze customer, but
I was put off by the abysmal desktop software :(

Storage execution seems top-notch. End-user software deserves some love.

~~~
bartvk
Abysmal compared to what?

I have experience with the Mac client, and it's rock solid, and very well
executed.

~~~
eiopa
The Windows client is bad. Can't speak about the osx one.

------
taneliv
I hope the memory is ECC, but don't see a mention in the article. Is it? At
least the CPU (and I think motherboard) support it.

~~~
atYevP
Rams -> Hynix 8GB PC3-12800 DDR3-1600MHz ECC Registered CL11 240-Pin

------
andrewbinstock
These posts set the standard for how hardware info should be communicated. Not
only the specs with elaborate detail and supporting images, the costs broken
down, and the rationale with explanations of what worked and what didn't.
Great stuff!

[edit: spelling]

------
lqdc13
Wow, all the components are crazy overpriced. You can get the same system
built for probably 1/3 the price with the same level of quality.

Example is the motherboard for $500+, the fans for $20 each, on off switch for
$25, 8gb of DDR3 ram for $90???? etc.

The saving grace is the hard drives that are decently priced and make up the
bulk of the cost.

~~~
budmang
Links? We love getting feedback, especially if we're overpaying for a
particular component?

Gleb

~~~
lqdc13
I guess it depends on the requirements a lot. Are you guys set on using ECC
RAM and microATX mobo? It would be much cheaper to use some sort of gaming cpu
+ DDR4 non-ecc RAM.

For example, microatx mobo:
[http://www.newegg.com/Product/Product.aspx?Item=N82E16813157...](http://www.newegg.com/Product/Product.aspx?Item=N82E16813157544)

some non-ecc DDR4 ram:
[http://www.newegg.com/Product/Product.aspx?Item=N82E16820231...](http://www.newegg.com/Product/Product.aspx?Item=N82E16820231796&cm_re=ddr4-_-20-231-796-_-Product)

and a cpu:
[http://www.newegg.com/Product/Product.aspx?Item=N82E16819117...](http://www.newegg.com/Product/Product.aspx?Item=N82E16819117402&cm_re=5820k-_-19-117-402-_-Product)

~~~
modoc
For a use like this EEC RAM is worth the money.

~~~
lqdc13
Alright, even with ECC and much faster DDR4 ram:
[http://www.crucial.com/usa/en/ct16g4rfd4213](http://www.crucial.com/usa/en/ct16g4rfd4213)

and motherboard:
[http://www.newegg.com/Product/Product.aspx?Item=N82E16813157...](http://www.newegg.com/Product/Product.aspx?Item=N82E16813157545)

~~~
lewisl9029
You might have missed the fact that the motherboard they use has built-in dual
10G Ethernet:
[http://www.supermicro.com/products/motherboard/Xeon/C600/X9S...](http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRH-7TF.cfm)

Once you account for the cost of a 10G NIC and the fact that having it
integrated frees up another expansion slot, I'd say it's a pretty good deal.

Also, I'm not sure if there are any non-Xeon _processors_ that support ECC.

~~~
ZoFreX
An extreme example of a non-Xeon that supports ECC: Atom C2750 :)

------
ausjke
so this is cheaper than even glacier? and it supports client-side-encryption?
and it will have a linux client (CLI is good enough)? if that's the case I'm
sold.

in fact do you have the storage located in various locations? if so can I
store a paid-copy to a separate location physically, just to be safe.

~~~
budmang
Just to be clear, there are 3 different things: * Backblaze Online Backup -
$5/month unlimited backup for Mac/Win; automatically does client-side-
encryption * Backblaze B2 Cloud Storage - $0.005/GB/mo for object storage
(lower cost than Glacier, but we don't yet offer a client, though others are
building) * Backblaze Storage Pod - the hardware that underlies both of these,
which we also open source for anyone to use

Gleb from Backblaze

~~~
gurkendoktor
> automatically does client-side-encryption

But server-side decryption :(

I have been looking into Backblaze every time it was mentioned on a podcast,
and as far as I can tell you still cannot download your backup and _then_
decrypt it on your own machine:

[https://en.wikipedia.org/wiki/Comparison_of_online_backup_se...](https://en.wikipedia.org/wiki/Comparison_of_online_backup_services#Comparison)

Such a pity. I'm still stuck with CrashPlan (yuck!) for this one reason. I've
been evaluating Arq, but I'm not yet confident enough to move all
my/relatives' machines over.

~~~
ausjke
looks like crashplan does support client-side encryption? is that true? does
it require a local mirror disk/folder like dropbox does?

I'm really more interested in a long-term backup storage instead of a folder-
sync storage, cold storage is fine but glacier seems a little hard to use and
retrieval is expensive.

------
amenod
Naive question: wouldn't it be possible with SSD drives (which are mainly just
chips, right?) to make the drives as boards and put them in 1U / 2U / 4U
chassis? I mean, who need the HDD dimensions when you are fitting the drives
inside server units anyway?

~~~
atYevP
Yev from Backblaze -> Not naive! You could definitely reconfigure the design
to hold SSDs, but that would add cost, which is something we're really
concerned about. But the short answer is yes, you can use SSDs, though it
increases the cost per GB.

------
amelius
I just posted a link to an interesting blog post detailing the risks of
building a storage pod [1]

[1]
[https://news.ycombinator.com/item?id=10541055](https://news.ycombinator.com/item?id=10541055)

~~~
acdha
Isn't that basically long-form restatement that you should know what you need
before spending money? The storage pods are pretty openly targeted at large-
scale operations where the lower storage costs pay for a dedicated ops team
and things like redundancy are handled at the software layer. Dinging them for
not having hardware RAID is like buying a semi-truck and complaining that it
doesn't fit in your garage.

~~~
aeorgnoieang
The post itself makes the same points.

~~~
acdha
Yes, but the headline and tone is different until you slog through to the
discussion & the comments making similar points.

------
mdamore
Great to see a smaller, very open company remaining competitive against the
giants in the space like Amazon.

I'm particularly excited to see how their B2 service will stack up to Glacier
for half the cost.

~~~
atYevP
Yev from Backblaze here -> We're pretty excited about it. Beta users are
providing some good feedback too! We think it'll definitely be the right
choice for a lot of folks!

~~~
thoughtpalette
Hi Yev! I commented on a previous post about Backblaze cloud storage and if it
was at a point to replace S3 for static assets.
[https://news.ycombinator.com/item?id=10259507](https://news.ycombinator.com/item?id=10259507)

"Is this going to work for static asset hosting such as for low cost FE
assets? Can this be set to serve an index file from the root directory? Doe's
it provide you a public URL to your (stealing s3 terms) bucket? Just curious
as to why I would migrate from S3 for FE assets to Backblaze."

Was wondering if any progress has been made since then, especially relating to
serving an index document from root dir.

Thanks!

edit: Awesome blog post btw

~~~
atYevP
Yes...we're making some ;-)

[https://www.backblaze.com/b2/cloud-
storage.html](https://www.backblaze.com/b2/cloud-storage.html)

------
qaq
or you could get Supermicro 6048R-E1CR72L with 72 drive-bays or something
similar for 3.5K and save yourself all the pain

------
ape4
Maybe by releasing the plans it will cause other folks to buy it and reduce
costs.

------
unixhero
Why are they not offering 8GB drives. It's late 2015.

~~~
atYevP
The most important thing for us is Cost per GB. Right now the 4TB drives we're
using are still much cheaper than the 8TBs, so we go with the 4s, but the Pods
work with 8TBs if you want to drop the dough on them!

~~~
beagle3
I buy ~20 drives/month (a drop in the bucket compared to you guys), but I find
6TB drives are fast, reliable, and cost the same per TB as the 4TB drives (so,
taking the NAS/Pod cost into account, cheaper per online TB).

Are 4TB drives significantly cheaper for you? Or is there some other
constraint at play against the 6TB?

------
ksec
This make, Google Drive, iCloud Looks expensive.

------
zimbatm
typo: "There are two Power Supplies, each with _their_ own wiring harness."

------
such_a_casual
It's wonderful what you guys are doing. Thank you for open sourcing your work.

~~~
atYevP
You're welcome! Our pleasure :)

------
hackaflocka
Local private cloud storage is going to be huge. Any new startups in this
space hiring?

~~~
atYevP
> Local private cloud storage is going to be huge

We're not really in that space, but we're always hiring :D

------
rasz_pl
>5 Port Backplane (Marvell 9715 Chipset) $43.85

Where exactly does one buy this?

------
therobot24
love the rundowns, sad that their service is just so disappointing

~~~
jakeva
I've used BB for years on multiple machines and have been more than satisfied,
in fact I recommend it to everyone. I'm not affiliated with them at all, just
a happy customer. Confused as to why someone would find their service
disappointing but to each their own.

~~~
therobot24
it takes months to get everything backed up, and now the just the file lists
or whatever is taking up 8GB+ on my disk (i use a smaller SSD as the primary
drive so space is precious). I contacted support and their only solution is to
create a new account and go through many more months of backing everything up
again.

~~~
josefresco
Does someone from BB want to jump in here and explain _why_ this guys "file
list" could be 8GB and how re-starting the sync will fix it? Sounds like
something went awry.

~~~
therobot24
This is what nathan from support told me when i contacted them:

"As part of our backup process, Backblaze will run a checksum against each
file before uploading it. This requires the entire bzfileids.dat file to be
loaded into RAM. After a long time, or if you have an extraordinarily large
number of files, the bzfileids.dat file can grow large causing the Backblaze
directory to appear bloated, and use more RAM (potentially slowing down your
system). In your case your bzfileids are extremely large, likely because you
have a lot of files and an older backup."

here's my backblaze folder:
[http://imgur.com/ohqUERr](http://imgur.com/ohqUERr) (i have my temporary data
drive set to a secondary drive)

