
Farming hard drives: how Backblaze weathered the Thailand drive crisis - zorlem
http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/
======
andrewcooke
what happened to the post that raised ethical questions about this? i think it
had a valid point.

the hard drive shortage hit us all. this one company coordinated a mass buying
operation from commercial stores when the rest of use were looking for
personal use.

while i don't think this is illegal, it doesn't make me feel warm and fuzzy
inside either.

when there's a shortage, the "noise" in traditional distribution means that
small buyers (like me, and many others) can still find what we need if we put
in some extra effort. it functions like a "soft buffer", making the transition
to a complete lack of resources more gradual. action like this removes that
buffer.

this feels like one group of people - driven by their desire to do business -
beat out a load of other people who didn't have the same logistical resources.
there's something uncool about it. i am not sure i have explained very well,
but it's to do with crossing the line from business to personal.

[edit: ah, thanks - separate thread:
<http://news.ycombinator.com/item?id=4630910>]

[edit2: wtf? i thought someone replied here? was about to upvote you. thanks
anyway.]

~~~
lusr
While I really enjoyed the story, I felt much the same way as you about the
ethics. They justify their decision to run business as usual, supporting the
same rate of natural growth, etc. by saying they didn't want to raise prices
for their customers, but I don't think that's really a fair reason (eminent
business failure would be a fair reason). There were other options:

1\. freeze registration of new accounts and purchase only the drives necessary
to keep existing contractual obligations

2\. charge new accounts at a different rate, promising to drop the rate once
drive prices returned to normal

3\. change plan storage volume limits

The only reason they didn't pick any of these 3 options, as far as I can tell,
is because all of them were bad for their profit margins and it would be nice
if they were upfront about that as the real driver for their actions. It's not
clear that their business would have failed had they chosen one of these
options.

So they chose a path that would screw over the little guy needing a new hard
drive in San Francisco. Perhaps I'm wrong, though, and the timing made it very
difficult to act on these other options, or their cashflow couldn't handle it.
That too would be nice to know, if it's the real driver of their actions.

Entirely rational behaviour (in the same position I'd do the same), but while
an entertaining story, it's a hard to story to feel enthusiastic about even
though in every other way it shows great leadership.

------
bruceboughton
The fact that external drives that have to be "shucked" of their casing and
circuit boards (at what environmental expense?) were cheaper than the same
internal drives makes you wonder whether the internal drive prices were
realistic.

It can't be cheaper to produce the external drives, so the internal drive
prices must have been raised above the actual cost to the manufacturers,
presumably because the internal drive demand was guaranteed. Consumers might
hold off purchasing an external disk if the prices doubled, but Apple aren't
going to stop selling Macs because of it.

Sometimes capitalism isn't perfect.

~~~
DanBC
Were they the same as the internal drives? Or just the same capacity?

I get the impression they just kludged as many 3 Tb drives in as they could,
and are ignoring any other specs. Maybe I'm missing something.

~~~
budmang
Gleb from Backblaze here. The drives were literally the same inside the
external enclosures. We bought a few to start with to check.

We use consumer-grade drives across our entire 40+ petabyte storage farm
because we do not need high-performance drives. Instead, we need low-cost,
low-power, high-density drives to put into our custom-designed Storage Pods:
[http://blog.backblaze.com/2011/07/20/petabytes-on-a-
budget-v...](http://blog.backblaze.com/2011/07/20/petabytes-on-a-
budget-v2-0revealing-more-secrets/)

We tested "enterprise drives" and found that from a pure reliability
perspective, they actually performed worse in our testing than the consumer
drives. With about 5 years and 40+ petabytes of data, we're fairly confident
this holds true.

Finally, we assume that all drives will fail and replace about 10 drives every
week. The software is written such that the system takes drive failures into
account and data is stored redundantly and continuously cross-checked.

~~~
stephengillie
It's interesting to find that "enterprise drives" performed worse than
consumer. Recently, a report came from a Microsoft datacenter, where they
found most errors weren't from faulty hardware, but firmware incompatibilities
between array controller and drive controller. I wonder if this type of
incompatibility occurs more often for enterprise drives than consumer.

------
rsync
We had this problem at rsync.net ... the price of our main consumable tripling
overnight, etc.

We really never considered "shucking" external drives, since the drives inside
these enclosures are the worst spec'd, highest failure rate drives ... never
mind the (usually) nonsensical enterprise vs. non-enterprise, they're just
crummy drives.

Good for BB for making it work ...

BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise
drives two months ago were $198 (or so) and last week they were $248 (or so).

~~~
mrb
But Blackblaze's infrastructure is properly designed to tolerate hw failures.
They use RAID6 and replicate your data to 2 or 3 different storage pods.
Smarter than beint forced to buy "better" expensive drives IMHO when your
primary business expense is raw storage.

~~~
luser001
If a disk fails, it still needs to be replaced. RAID etc lets you avoid data
loss _until_ you replace the drive.

------
bruceboughton
If BackBlaze are adding 50TB of data a day, are they really buying drives from
NewEgg?

~~~
michaelt
A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I
could order them individually from Amazon for £27 each with free delivery.

I phoned up all the official Microsoft distributors in my country, who phoned
up Microsoft and whatnot. Not one of them could offer me a price below about
£35. And getting in contact with Microsoft hardware directly? Ha, I should be
so lucky.

~~~
andrewf
_A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams._

I'll bite. Why? :)

~~~
michaelt
Asked to quote to make a system like [1] for a fleet of 1000 vans. Buying in
USB cameras would reduce engineering costs compared to making our own.
Eventually went for a brand that cost less and could accept wide angle lenses.

[1] <http://www.smartdrive.net>

------
yock
If I were their customer, I would be looking for alternatives right about
now...

~~~
aes256
The post struck me as slightly unprofessional, but from a technical standpoint
I don't see anything that would change my opinion of the service they provide.

They achieve the best bang for buck by using ordinary consumer hard drives in
RAID arrays. During the hard drive crisis, the cheapest way of procuring these
was by purchasing external drives from ordinary stores.

There's nothing wrong with that. If anything, it shows agility and ingenuity.

~~~
yock
And, I would add, intense vulnerability to future market fluctuations. If this
was their contingency plan, they've exhausted it. What happens next time?

~~~
aes256
This was a one-off event that I'm sure the hard drive manufacturers themselves
will be taking steps to prevent reoccurring.

What else can Backblaze do? Stockpile drives? The hard drive industry moves
fast, with major product releases every 6-12 months and constant price
reductions; this would almost certainly increase the cost to the end user.

No business can adequately prepare for a supply shock like this. Imagine
airlines facing a 300% increase in the price of fuel. Almost every single one
would go out of business.

In this case, would you be berating the ingenious airline that goes out of its
way to track down fuel at a cheaper price, finding a low-cost fuel that they
can easily refine themselves to aviation fuel, so as to honour their
commitments to their customers and avoiding raising their prices?

~~~
anonymouz
> Imagine airlines facing a 300% increase in the price of fuel. Almost every
> single one would go out of business.

No. They use hedges to control their cost (this is the original point of
hedging...). If a company is reliant on a constant supply of hard-drives and
doesn't want to be exposed to the risk of extreme price jumps it should
probably do the same.

~~~
jws
Airlines have long term fuel contracts to smooth out the short term price
changes, but beyond that they just pass it on to consumers.

e.g.
[http://www.delta.com/business_programs_services/delta_cargo/...](http://www.delta.com/business_programs_services/delta_cargo/current_surcharges/index.jsp)
the first column is the fuel surcharge that floats with fuel prices.

The Backblaze article clearly said that paying more for drives and passing it
on the customers was an option. They just found a better one. They didn't
mention it, but just losing money for a while is also a reasonable option if
you don't want to antagonize your customers.

~~~
atYevP
*Yev from Backblaze here -> You're correct, but quick note on this. We're a boostrapped startup, meaning, we need to be constantly profitable in order to pay salary and keep the business going, taking a financial hit was the only real option we could not take, because we would have been out of business very quickly (not good for anyone).

------
pella
same news ; and comments:

<http://news.ycombinator.com/item?id=4630910>

~~~
damncabbage
This one is the original article, and is better-written too.

(Gigaom blogspammed it.)

------
w1ntermute
I wonder if the hard drive manufacturers will reconsider locating their entire
supply chain in a third world country with frequent natural disasters after
this incident. With rising wages in the third world and stagnating wages in
America, perhaps the financial benefits of outsourcing aren't so apparent
anymore.

~~~
rogerbinns
They have more data about all the pieces and processes leading to their
ultimate market share and profitability so I have no doubt they try to
maximize those.

I suspect the biggest issues are that all their suppliers and customers are in
that region. Think fabs to make the chips, the various motors and related
electronics, coating processes, and then sending them off to the places that
make HP/Dell/etc systems. If just the drive factory were moved then you'd
still have the drive parts crossing the ocean to get to the US and the
completed drives crossing once again to go back into the systems.

Usually the relocations happen when a disruptive solution comes along. You get
the advantage of a somewhat clean sheet. For example SSDs don't need all the
mechanical components and fine tuning, so they can be made with a completely
different supply chain.

It won't happen in hard drives but for other products customization is what
can spur relocation. For example Lenovo is opening a manufacturing plant in
the US to deal with customization which allows quicker response times.
(Customized orders often can charge a little more which offsets increased
costs.)

------
abalone
Now that's a startup story. Backblaze has some of the best down home old
fashioned spit n glue startup stories on hacker news. Like the one where they
invented their own storage pod out of commodity components to beat amazon's
pricing. And now this stay-alive-against-all-odds story. Fantastic.

------
granite
I was impressed with the effort to get those drives. If the massive companies
who produce hard drives were unprepared for such a natural disaster I can't
say I expect more from a small company.

------
JakeFratelli
Their execution was poor. Start by depleting SF and then look elsewhere?
Short-sighted at best.

They should have moved quickly before Best Buy/Costco implemented constraints
that should have been anticipated. During one weekend, via train or one-way
flights, employees could have been mobilized to cities across the US.
Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have
been shipped back to HQ en masse. Three days, disaster is over.

I give them a C- on execution.

~~~
rtkwe
That's predicated on them being able to spare many employees just for drive
sourcing, which as a relatively small company they probably couldn't. They
went with the lower cost stop gap at first, if the shortage hadn't lasted as
long they would have been better off with how they did it.

