

How a few screws cost $2000 and a 240GB multinodes cluster cost $50 - plam
http://www.quantisan.com/how-a-few-screws-cost-2000-and-a-240gb-multinodes-cluster-cost-50/

======
shinratdr
There are countless flaws in this comparison, but the one that stands out to
me is the simple buying vs renting issue. They sourced screws that would be
permanently installed into something. They rented a couple hours on a server.

Renting is always cheaper than owning. That's a pretty universal rule.
Complain about the red tape all you want, I have also felt the burn of the
"approved supplier" list. That doesn't excuse an extremely faulty comparison,
however. It just clouds the issue.

I understand coming up with a better comparison would be a little more work
than the immediate but completely faulty on-hand cost of the server instance
they spun up, but I know the cost of the computer I own and the cost of a
server I rent and I wouldn't use them to make a comparison just because I'm
too bothered to come up with a real comparison.

There is also the issue of the fact that the $2000 number was pulled out of
seemingly thin air, the possibility that they use approved suppliers to avoid
the issue of counterfeit or sub-standard parts, which have killed people when
installed into regular things like bleachers that stay on the Earth. It's the
freaking Canadarm, god forbid we splurge now so we don't waste millions later
fixing it or kill someone. When the end result of the materialized idea is
it's going to be launched into space, the costs probably aren't so black and
white.

Interesting story, but with so many variables unaccounted for I have a hard
time gleaning any sort of lesson from it. Other than the fact that building a
piece of equipment that will be launched into space involves a fair bit of red
tape, which I was already aware of and am honestly kind of thankful for.

~~~
icebraining
_When the end result of the materialized idea is it's going to be launched
into space_

Yes, but OP clearly says the boxes were never to be used in production, only
testing.

~~~
mariuolo
I'm sure I don't have to tell you how much of a bad idea is to use testing
environments not matching production, for something to be put into space on
top of that!

~~~
bigiain
Did you not read the article? This was a simulated load testing device that
they could plug production components into to to experiment on them without
needing to have the entire space-going arm on hand and powered up. Suggesting
that a test component like that needs to be built to the same standards as the
space-going equipment is, ummm - how do I out this politely? Wrong…

~~~
druiid
I disagree here. While it is a testing component, part of any valid test is
that it be replicable. If you are running down to 'Home Depot' and picking up
some random screws with who knows what specifications, how exactly are you
going to replicate the design here? Perhaps they will never need to do it and
it is a genuine one-off design, but putting on the line a potential multi-
million (billion?) dollar project? In the grand scope then, spending $2000
(which I doubt WAS the real cost here) on some screws just gets a _shrug_ out
of me.

~~~
jevinskie
The safety cover is not part of the device under test and I doubt it interacts
with it except during a catastrophic failure. It does not need to be
replicated exactly the same in a future test.

------
CognitiveLens
The comment backlash against this anecdote is a little odd to me. No, it's not
a perfect analogy. No, the cost comparison is neither fair nor complete nor
comprehensive.

But it _is_ an interesting connection to make, and there _are_ interesting
ideas that the comparison generates, so let's treat it like a parable rather
than a critical business analysis.

If anything, one of the most striking comparisons is how work was done in a
government-funded research lab versus how it is done in a startup. Own vs
rent, meticulous simulation vs rapidly-iterated prototyping, waterfall vs
agile, regulation vs free for all.

The superficial details of the story aren't actually that important, but
there's a lot of meat to dig into.

~~~
rayiner
People are complaining because the author ignores his own take-away:
"Different time, different industry."

All of this "agile," "free for all," "start up" cuteness is okay when you're
dealing with software, which generally only ever results in frustration or at
worst lost profits. The standards and practices in industries where things can
go wrong in epic, expensive, and dangerous ways is obviously quite different.

------
DanBC
The first example includes cost of labour. The second example doesn't.

Two people playing with that cluster for a couple of hours cost more than $45.

Still, the author is right that when building physical products sometimes
things are a lot more expensive than you think they're going to be because
you're following some strict protocol.

~~~
cnvogel
Yes, you are absolutely right. But I think this is not the main point of the
article.

The main point is that in a huge corporation with rules for purchasing and
choosing suppliers, there's an insane amount of labor spent in administration
even for purchasing even the most mundane things, as soon as it's something
deviating from "the norm".

I've worked in a highly regulated industry, in a pretty big company, and my
experience was the following: If you always order €1000 devices from a
company, this company is registered in the internal databases, and had an
audit from your quality assurance company. All traffic-lights will be green.
You order the device, you, your boss, the financial responsible or whatever
signs the document and they will bill one hour of work for that single-hand-
move operation, which will be in the order of €100/$100. Total cost for your
department of project: €1100 for tie €1000 device.

Now if you are in the lab, and think it's cool to use that Arduino shield for
$10 on seedstudio for your lab-setup (which is probably a completely
reasonable approach, as you only want to turn on/off that light/motor/... you
normally would have to do manually for your experiment/measurement/test) the
pain begins:

Seedstudio is not on the list of approved vendors, the QA team has to be
persuaded that this is not a item that will ever reach a customers
installation so they don't travel there to do an audit of the ISO9000
compliance (or whatever special regulation your partícular industry demands).
Purchasing will complain that seedstudio has never signed your "terms of
order" (and they will just throw away the 10-page-legalese anyway for $10...).
And controlling will complain endlessly about the "proper inventory code" for
a "seedstudio stepper motor shield". They'll also give you a sticker and make
it a "IT inventory item" because it can be connected to a PC. In the end
you'll have 5 people bill each 2 hours on that issue, and your project will be
billed 5 _2_ $100 + $10 = 1010$ of the Arduino shield. And you'll be known as
"that hacker guy" that "constantly orders strange things from chinese
companies" through the whole company.

Everyone new to that kind of big corporation tries this once, or twice, and
then decides that it's just not worth it. In practice, you'll buy it on your
private credit card, and hand it in as "travel expenses" on your next business
trip. Or you'll have subcontractor already working with you that will order it
for you, forward the device, and just put "Misc. manufacturing material. $50"
on the next monthly $20'000 bill that on one will ever check.

~~~
jzwinck
So, fight inefficiency with fraud. Got it.

Each of those arcane rules was probably put in place because someone a long
time ago did something they weren't supposed to do. "Working the system" by
intentionally ignoring the rules begs two questions: who needs these rules,
and how long until you get caught and more rules are piled on for the next guy
to get jaded about?

At some point it becomes madness. Sometimes that's when your company gets its
lunch eaten.

~~~
CaptainZapp
Well, if you see it from the perspective of a corporation then procurement is
probably one of the fields most ripe for fraud.

If you're just able to hand in an invoice to accounts payable then it's very,
very easy to come up with bogus invoices. In fact this used to be a quite
common scam.

Thus those rigid controls on all things procurement.

I worked for a variety of global companies and government entities. Setting up
the whole invoicing process is generally a pain and usually you'll have to
wait for a couple of month until the process works flawless.

Once it works it usually does work flawless.

I'm not arguing that this is great. But I can understand where a company is
coming from, since the whole field of procurement is just a huge invite for
fraud and abuse, if it's not rigidly controlled, which, I'm afraid, also
involves an approved supplier list.

[EDIT : Spello corrected]

------
rayiner
Ordering from an approved supplier with a traceable chain of delivery for
equipment used in a space program seems quite reasonable to me. There is a
reason that people from the more traditional engineering disciplines
(aerospace, chemical, civil, mechanical) are so anal-retentive about "proper
engineering guidelines." It's because they deal with things that blow up, and
when they do blow up shoot turbine blade shrapnel all over the place. Nearly
everyone working in these fields has some story about how a little screw up in
one out-of-the-way component caused some epic disaster that involved fire.

Yeah, sure in this case it was just some screws on a plastic housing and there
was a 99.999% chance that everything would've been fine. But you don't want to
make exceptions, because if you play fast and loose here and there, eventually
all the prudent practices go out the window and you're no better than the
software guys.

~~~
olympus
< _and you're no better than the software guys._ >

This is a huge point that many HNers probably don't get(but they think that
they get). Releasing patches/bugfixes doesn't work so well on physical stuff.
If Amazon AWS goes down, a group of people can figure out what went wrong, fix
the issue and "reboot" the system and all that is lost is a bit of revenue and
a portion of their reputation. When pieces break on heavy iron you don't have
the time to figure out what went wrong and then fix it because someone is
about to crash. Hence the difference between the traditional engineers and the
software folks is the traditional engineers have to fix stuff before it
breaks, software engineers can afford to fix stuff after it breaks.

~~~
rayiner
> Hence the difference between the traditional engineers and the software
> folks is the traditional engineers have to fix stuff before it breaks,
> software engineers can afford to fix stuff after it breaks.

Absolutely. The mindset is just very different. I remember once running some
software in debugger during a major tech demo, and fixing a display bug I saw
was going to become a problem in the next few minutes using Edit & Continue in
Visual Studio. That's the "software guy," "start up," way of doing it. That
shit would never fly with "hardware guys."

------
omegaham
This happens all the time in government. Business has the lucky position of
constantly scrambling to cut costs, cut redundancy, cut inefficiency.
Otherwise they'll be beaten by a new company that isn't tied down by self-
induced paralysis. The government has no such incentive to do so. When faced
with a choice between saving fifty billion dollars and having the _assurance_
that everything is hunky-dory, the government will spend the fifty billion
dollars without a second thought.

One of my duties, along with being a radio technician, is ordering replacement
parts for the systems that we work on. Some of the stuff is expensive and
reasonably so - for example, a circuit board with $200 bucks in components is
$20,000 because it has to be professionally made and is basically a one-of-a-
kind part. The supplier has to charge that just to recoup the costs of
retooling. We accept this because we need very reliable parts on radar systems
and are willing to pay for the assurance that these parts will be good.

On the other hand, I just ordered an audio cable through the FAA. Simple two-
wire audio cable... $333.45. I could've bought the exact same thing from
Radioshack for five bucks. Or Monoprice for one dollar.

At one point, Raytheon told us that we were not allowed to do any
intermediate-level maintenance (anything involving fixing circuit cards) on
their stuff. We were expected to send the bad cards to them, and they would
give us replacements and bill us for $10,000 each. The chief warrant officer
laughed, said "Fuuuck no," and told us that if we could easily fix it, do so.

Raytheon realized we weren't sending them any circuit cards and called the
commanding general to get him relieved. He didn't get fired, but we started
sending the cards to them for broken 2-cent resistors. For ten grand each.
Same thing with $30,000 power supplies, etc.

Right now, the government has declared that getting office supplies from
Staples is horrible. We're supposed to get them from Servmart, which sells
them for five times the price... or more. Cheap-shit ballpoint pens for a
dollar each when I can get Bics for $4.60 for 72 of them.

Your tax dollars at work, gents.

------
ctdonath
Screws for spacecraft vs borrowing a machine to play with, ok. $2000 vs $45.

Someone explain why a frying pan and a Raspberry Pi cost the same.

~~~
shubb
Wow - That is a really amazing question. I'll be wondering about it for a
while...

The following is speculation.

On AliBarBar, you can find wholesale pans at source in china for $1 each. The
price of pi like devices is about $30 dollars.

So, why do pans get so much more expensive as they cross the ocean?

1\. Cost of postage

2\. Protectionist import taxes. Here is Europeans list of things they are
thinking of levvying punative taxes on. It's long.
<http://trade.ec.europa.eu/tdi/notices.cfm>

3\. Regulatory approval. The burden of bringing a new electronic device to
market is pretty high. You need specialist facilities (or to pay a lot of
money to a test lab) to prove electrical emissions are fine. This is why so
many Chinese designs never make it to the UK. Alibaba is full of lovely
android powered arm net-tops, but you note the ones you see in electronic
stores are from the same limited range, even the cheap ones. This is why.

I presume the same is true for pans. So although designing a pan and bringing
it to manufacture is simpler than manufacturing a new ASIC or even embedded
computer, only a very few pan designs can be sold in the UK. It just isn't
worth bringing another one through approval (unless it's much cheaper). And
the approved designs can sell for much more.

One hears about western companies keeping as much of the 'value add' as
possible out of China and India, possibly because it's so expensive to
retrieve money once it's in there. I wonder if this also is part of it.

~~~
bigiain
"So, why do pans get so much more expensive as they cross the ocean?"

My guess? Because pans get sold by experienced companies who recognize that
the price of an item is decided by its utility and the amount people are
prepared to pay for that utility, not anything to do with the cost of
manufacture. The gap between the manufacturing cost and the utility value of
an item only lets you choose whether or not to be in the business of selling
that item.

~~~
AnthonyMouse
>Because pans get sold by experienced companies who recognize that the price
of an item is decided by its utility and the amount people are prepared to pay
for that utility, not anything to do with the cost of manufacture.

The price of an item is decided by supply and demand. The only way you get a
retail price which is significantly higher than the manufacturing cost is a
lack of competition.

For items like pans this is probably more at the retail level than the
manufacturers. I kind of doubt Walmart is buying $1 pans for $45 and selling
them for $50.

~~~
bigiain
No, I think this is where the "pan selling industry" and the "tech toy/trinket
selling industry" are different.

There's something going on in the "pan selling industry" which results in very
little "race for the bottom price" type competition - there are clearly many
individual companies importing pans from China, but the "well known retail
price" of those pans is ~$30 no matter which retailer you buy it from (within
reason - at least within enough reason that all the $30 price-point retailers
are still in business).

Tech toys, on the other hand, quite often end up in the "race to the bottom"
price category - with many innovative and desirable items ending up available
from sites like Alibaba or DealExtreme at ludicrously low prices.

I think part of it is the audience - the "pan buying demographic" is probably
a lot earlier in it's uptake of globalised web based purchasing than the
demographic who's buying Arduinos and RasberryPis. Perhaps one day your
average mid-western housewife will say "I need a new saucepan, I'll just check
Alibaba before I drive to the mall and pick one off the shelf at Walmart - if
I can save $20 and only need to wait a week I'll just get it delivered."

But like I said - I suspect it's "experienced companies" who are not exactly
colluding or price-fixing, but who all know that it's better for _all_ of them
to keep pans priced at ~$30 and have everybody make margins of almost $29 per
sale, rather than end up trying to compete on price at $5/pan knowing that the
total market for pans won't increase just 'cause the prices drop 85%.

(And sometimes, you get the evil-Walmart-effect, where someone big enough to
absorb the temporary losses chooses to drop the consumer-expected pricing down
to $2/pan, waits out until the competition goes broke and vanishes, then puts
the prices back up near $30/pan while owning all or most of the market.)

~~~
AnthonyMouse
>There's something going on in the "pan selling industry" which results in
very little "race for the bottom price" type competition - there are clearly
many individual companies importing pans from China, but the "well known
retail price" of those pans is ~$30 no matter which retailer you buy it from
(within reason - at least within enough reason that all the $30 price-point
retailers are still in business).

That's what I'm saying -- they don't compete with each other. There isn't
enough competition that anyone decides to try to improve their market share by
lowering prices, because it's just Bed, Bath & Beyond and Sears and such like
"competing" with each other in any given local area, and not enough people buy
housewares over the internet to move the needle against those guys.

>I think part of it is the audience - the "pan buying demographic" is probably
a lot earlier in it's uptake of globalised web based purchasing than the
demographic who's buying Arduinos and RasberryPis.

More than that, electronics are purchased in bulk by corporate purchasing
departments. If you're buying 25 new devices every two years you may not be
such a big fish that you qualify for a volume discount, but you're certainly
spending enough money to justify a significant amount of comparison shopping,
which means a high price elasticity of demand and an immediate market share
advantage for retailers who engage in price competition.

>But like I said - I suspect it's "experienced companies" who are not exactly
colluding or price-fixing, but who all know that it's better for _all_ of them
to keep pans priced at ~$30 and have everybody make margins of almost $29 per
sale, rather than end up trying to compete on price at $5/pan knowing that the
total market for pans won't increase just 'cause the prices drop 85%.

The phrase you're looking for is "conscious parallelism." You get de facto
collusion without communication by means of everyone adopting the strategy
that they not be the first to engage in price competition. It works only so
long as you have sufficiently few competitors that all of them religiously
follow the strategy -- because having a third of the market with 5X the
margins is more profitable than having the whole market at 1/5th the margins.
The problem comes when you have more competitors, in which case someone
decides they would rather have half of the market at 1/5th the margins than
1/15th of the market at 5X the margins, and believes (perhaps correctly) that
they will be the one to survive a price war. This has basically been Walmart's
business model for probably the majority of the products they sell, and Amazon
et al are now doing the same thing even more aggressively on the internet.

------
guelo
This is why as an intern at NASA I decided government work wasn't for me and I
eventually ended in startup world. However, I think there is something to be
said for creating within strict guidelines and procedures, it can sometimes
lead to amazing engineering achievements.

------
pbo
This also shows in how quick you can iterate. The "REPL time" in mechanical
engineering has decreased a lot with CAD software but it's still insanely long
compared to software development.

I used to work in a lab with a researcher who designed complex mechanical
systems with lots of rotating parts made from rare and expensive materials.
Everytime he needed even the slightest modification he had to wait for at
least three months to have a new part machined.

------
StavrosK
How a comment cost $10,000 and a car cost $5: I got you all to read this,
wasting your time and electricity, computer costs, etc. In contrast, I got a
cab yesterday and it only cost $5.

I don't generally insist that everything posted be transcendent, but I fail to
see even a fair point being made in the post. If the point is "buying things
for an aerospace corporation is more expensive than renting an hour on EC2",
okay, but we sort of knew that already.

------
starky
I work in hardware, and wanted to give my perspective on this.

Designing and building hardware is expensive, really expensive. If you don't
take the time to follow proper procedures you are not only wasting your time,
but often hundreds to thousands of dollars of parts you are never going to
use. Going ahead without thinking can easily put you 2 or more weeks behind
just because that is how long it takes your supplier to make what you ordered.
You don't just quickly spin up an instance of a test, you have to buy or rent
equipment that is often in the thousands to hundreds of thousands of dollars
range.

I can see exactly why you wouldn't want to just go to Home Depot when working
on equipment like what MDA makes. Say you go to Home depot and buy your
fasteners, now the company has to pay for your time and reimburse the cost,
which is almost certainly going to be as much or more to process than the
courier charge ($30-$40) to ship the parts from the company you are making
monthly payments to regardless. Then there is the additional risk of bringing
in parts that could get mixed up with real production parts. There is a huge
risk if you somehow get that Home Depot fastener mixed up with the special
aerospace fastener you are using, the extra cost is worth it as insurance that
"bad" fasteners aren't getting into the product. I can say from experience, it
is ridiculously easy to get parts mixed up even when you are careful.

~~~
Dylan16807
I fail to see how filling out $2000 in paperwork will stop you from getting
screws mixed up. Because that's what the cost was, apparently. Not the actual
equipment.

------
kfk
Those rules are important. Not every company in the world is a start up with
<100 people. Once you have thousands of employees, you need controls. You
can't have interns selecting suppliers or putting stuff into the bill of
materials, are you serious with this?

------
FollowSteph3
Unfortunately we're just as much to blame for the red tape. It didn't get
created out of a vacuum, it was created to protect people. I wrote a whole
blog post called Why Does a Hammer Costs $5000 at
[http://www.followsteph.com/2011/11/22/why-does-a-hammer-
cost...](http://www.followsteph.com/2011/11/22/why-does-a-hammer-cost-5000/)

------
dschiptsov
why, in Russia, on government sponsored Olympic village construction projects
$2k for a few screws is strikingly cheap.)

