Hacker News new | comments | show | ask | jobs | submit login
How a few screws cost $2000 and a 240GB multinodes cluster cost $50 (quantisan.com)
85 points by plam 1446 days ago | hide | past | web | 46 comments | favorite



There are countless flaws in this comparison, but the one that stands out to me is the simple buying vs renting issue. They sourced screws that would be permanently installed into something. They rented a couple hours on a server.

Renting is always cheaper than owning. That's a pretty universal rule. Complain about the red tape all you want, I have also felt the burn of the "approved supplier" list. That doesn't excuse an extremely faulty comparison, however. It just clouds the issue.

I understand coming up with a better comparison would be a little more work than the immediate but completely faulty on-hand cost of the server instance they spun up, but I know the cost of the computer I own and the cost of a server I rent and I wouldn't use them to make a comparison just because I'm too bothered to come up with a real comparison.

There is also the issue of the fact that the $2000 number was pulled out of seemingly thin air, the possibility that they use approved suppliers to avoid the issue of counterfeit or sub-standard parts, which have killed people when installed into regular things like bleachers that stay on the Earth. It's the freaking Canadarm, god forbid we splurge now so we don't waste millions later fixing it or kill someone. When the end result of the materialized idea is it's going to be launched into space, the costs probably aren't so black and white.

Interesting story, but with so many variables unaccounted for I have a hard time gleaning any sort of lesson from it. Other than the fact that building a piece of equipment that will be launched into space involves a fair bit of red tape, which I was already aware of and am honestly kind of thankful for.


When the end result of the materialized idea is it's going to be launched into space

Yes, but OP clearly says the boxes were never to be used in production, only testing.


I'm sure I don't have to tell you how much of a bad idea is to use testing environments not matching production, for something to be put into space on top of that!


Did you not read the article? This was a simulated load testing device that they could plug production components into to to experiment on them without needing to have the entire space-going arm on hand and powered up. Suggesting that a test component like that needs to be built to the same standards as the space-going equipment is, ummm - how do I out this politely? Wrong…


I disagree here. While it is a testing component, part of any valid test is that it be replicable. If you are running down to 'Home Depot' and picking up some random screws with who knows what specifications, how exactly are you going to replicate the design here? Perhaps they will never need to do it and it is a genuine one-off design, but putting on the line a potential multi-million (billion?) dollar project? In the grand scope then, spending $2000 (which I doubt WAS the real cost here) on some screws just gets a shrug out of me.


The safety cover is not part of the device under test and I doubt it interacts with it except during a catastrophic failure. It does not need to be replicated exactly the same in a future test.


Yeah sure - but where do you stop? Where do you draw the line and say "this bit doesn't need to be space-rated"?

Reductio ad absurdum says the table it sits on is "part of the test", and the power supply it runs off is "part of the test", and the engineers car that he uses to get to work to run the test is "part of the test"...

I'd _strongly_ advise that a screw that hold together the enclosure of a load-simulating-test-jig is _not_ "part of the test" in the way that requires ISO9000 compliance (or space-rating).


I don't think the OP ever said it was a mistake. He said he wouldn't make that choice, but that doesn't mean he thought it was wrong for the company.

Instead he contrasted it to what you can do today as a startup.


> Suggesting that a test component like that needs to be built to the same standards as the space-going equipment is, ummm - how do I out this politely? Wrong…

What would be the point of testing it then? If the test isn't replicating the production environment before production, what is the point? At that point you might as well just hot-glue the safety cover on. What's the difference?


That's my point - who's testing the mechanical characteristics of the retaining fasteners of the enclosure that the electronics load simulating is housed in? Is that part of any spec? Is it at all relevant to the success/failure of any of the space-rated or mission critical equipment?

Me, I'd _happily_ hot glue the cover on if it were more convenient to me at the time. I see this as mostly a workplace safety issue - the screw holding the cover on is _not_ "part of the test", it's mostly to keep accidental fingerpoking out.


RTFA. The reason that they made the decision to follow process was because they chose to apply the same procedures to everything to avoid picking and choosing which items were exempt.


If you start making exceptions you'll end up wasting more resources to keep track of them and evaluating whether they're acceptable for the scenario than just not making any as general rule.


...isn't that the point of testing? To determine what should be used in production?


Another aspect is that cloud suppliers have way more customers to distribute fixed costs on compared to aerospace firms.


Not yet. Aerospace firms providing fasteners have a huge number of customers and it's a competitive industry. They also have clients in countries that have no Internet access to speak of.


The comment backlash against this anecdote is a little odd to me. No, it's not a perfect analogy. No, the cost comparison is neither fair nor complete nor comprehensive.

But it is an interesting connection to make, and there are interesting ideas that the comparison generates, so let's treat it like a parable rather than a critical business analysis.

If anything, one of the most striking comparisons is how work was done in a government-funded research lab versus how it is done in a startup. Own vs rent, meticulous simulation vs rapidly-iterated prototyping, waterfall vs agile, regulation vs free for all.

The superficial details of the story aren't actually that important, but there's a lot of meat to dig into.


People are complaining because the author ignores his own take-away: "Different time, different industry."

All of this "agile," "free for all," "start up" cuteness is okay when you're dealing with software, which generally only ever results in frustration or at worst lost profits. The standards and practices in industries where things can go wrong in epic, expensive, and dangerous ways is obviously quite different.


thank you for understanding and clarifying my intent


The first example includes cost of labour. The second example doesn't.

Two people playing with that cluster for a couple of hours cost more than $45.

Still, the author is right that when building physical products sometimes things are a lot more expensive than you think they're going to be because you're following some strict protocol.


Yes, you are absolutely right. But I think this is not the main point of the article.

The main point is that in a huge corporation with rules for purchasing and choosing suppliers, there's an insane amount of labor spent in administration even for purchasing even the most mundane things, as soon as it's something deviating from "the norm".

I've worked in a highly regulated industry, in a pretty big company, and my experience was the following: If you always order €1000 devices from a company, this company is registered in the internal databases, and had an audit from your quality assurance company. All traffic-lights will be green. You order the device, you, your boss, the financial responsible or whatever signs the document and they will bill one hour of work for that single-hand-move operation, which will be in the order of €100/$100. Total cost for your department of project: €1100 for tie €1000 device.

Now if you are in the lab, and think it's cool to use that Arduino shield for $10 on seedstudio for your lab-setup (which is probably a completely reasonable approach, as you only want to turn on/off that light/motor/... you normally would have to do manually for your experiment/measurement/test) the pain begins:

Seedstudio is not on the list of approved vendors, the QA team has to be persuaded that this is not a item that will ever reach a customers installation so they don't travel there to do an audit of the ISO9000 compliance (or whatever special regulation your partícular industry demands). Purchasing will complain that seedstudio has never signed your "terms of order" (and they will just throw away the 10-page-legalese anyway for $10...). And controlling will complain endlessly about the "proper inventory code" for a "seedstudio stepper motor shield". They'll also give you a sticker and make it a "IT inventory item" because it can be connected to a PC. In the end you'll have 5 people bill each 2 hours on that issue, and your project will be billed 52$100 + $10 = 1010$ of the Arduino shield. And you'll be known as "that hacker guy" that "constantly orders strange things from chinese companies" through the whole company.

Everyone new to that kind of big corporation tries this once, or twice, and then decides that it's just not worth it. In practice, you'll buy it on your private credit card, and hand it in as "travel expenses" on your next business trip. Or you'll have subcontractor already working with you that will order it for you, forward the device, and just put "Misc. manufacturing material. $50" on the next monthly $20'000 bill that on one will ever check.


So, fight inefficiency with fraud. Got it.

Each of those arcane rules was probably put in place because someone a long time ago did something they weren't supposed to do. "Working the system" by intentionally ignoring the rules begs two questions: who needs these rules, and how long until you get caught and more rules are piled on for the next guy to get jaded about?

At some point it becomes madness. Sometimes that's when your company gets its lunch eaten.


Well, if you see it from the perspective of a corporation then procurement is probably one of the fields most ripe for fraud.

If you're just able to hand in an invoice to accounts payable then it's very, very easy to come up with bogus invoices. In fact this used to be a quite common scam.

Thus those rigid controls on all things procurement.

I worked for a variety of global companies and government entities. Setting up the whole invoicing process is generally a pain and usually you'll have to wait for a couple of month until the process works flawless.

Once it works it usually does work flawless.

I'm not arguing that this is great. But I can understand where a company is coming from, since the whole field of procurement is just a huge invite for fraud and abuse, if it's not rigidly controlled, which, I'm afraid, also involves an approved supplier list.

[EDIT : Spello corrected]


Indeed, a better way to fight this would be to start making the cost visible to your manager. Just start a spreadsheet, "item cost" vs "total company cost", resulting in "Cost overhead". Just hand these to your boss each month. No snarky comments, no aggression, just "Look, I have gathered some interesting information".

Once he sees a rather ridiculous cost overhead right there on paper and ideally even verfiable with your orders through other systems, interesting things happen. More often than not, management didn't know the excess of the situation.


We have department P-cards where we can buy small stuff from pretty much anyone who accepts credit cards, I think we can even do PayPal now. For large purchases we have to use approved vendors and there's red tape but it saves the company billions of dollars (literally, where I work).

In fact, in most large electronics companies the real money is made and lost in the supply chain so they have incredible power, annoying as it sometimes is.


The counterpart to the cost of labour from the screws story would be the simple "click" from the server story.

The cost and value of using the bought products (screws/servers) is not included in either example.


Ordering from an approved supplier with a traceable chain of delivery for equipment used in a space program seems quite reasonable to me. There is a reason that people from the more traditional engineering disciplines (aerospace, chemical, civil, mechanical) are so anal-retentive about "proper engineering guidelines." It's because they deal with things that blow up, and when they do blow up shoot turbine blade shrapnel all over the place. Nearly everyone working in these fields has some story about how a little screw up in one out-of-the-way component caused some epic disaster that involved fire.

Yeah, sure in this case it was just some screws on a plastic housing and there was a 99.999% chance that everything would've been fine. But you don't want to make exceptions, because if you play fast and loose here and there, eventually all the prudent practices go out the window and you're no better than the software guys.


<and you're no better than the software guys.>

This is a huge point that many HNers probably don't get(but they think that they get). Releasing patches/bugfixes doesn't work so well on physical stuff. If Amazon AWS goes down, a group of people can figure out what went wrong, fix the issue and "reboot" the system and all that is lost is a bit of revenue and a portion of their reputation. When pieces break on heavy iron you don't have the time to figure out what went wrong and then fix it because someone is about to crash. Hence the difference between the traditional engineers and the software folks is the traditional engineers have to fix stuff before it breaks, software engineers can afford to fix stuff after it breaks.


> Hence the difference between the traditional engineers and the software folks is the traditional engineers have to fix stuff before it breaks, software engineers can afford to fix stuff after it breaks.

Absolutely. The mindset is just very different. I remember once running some software in debugger during a major tech demo, and fixing a display bug I saw was going to become a problem in the next few minutes using Edit & Continue in Visual Studio. That's the "software guy," "start up," way of doing it. That shit would never fly with "hardware guys."


Screws for spacecraft vs borrowing a machine to play with, ok. $2000 vs $45.

Someone explain why a frying pan and a Raspberry Pi cost the same.


Wow - That is a really amazing question. I'll be wondering about it for a while...

The following is speculation.

On AliBarBar, you can find wholesale pans at source in china for $1 each. The price of pi like devices is about $30 dollars.

So, why do pans get so much more expensive as they cross the ocean?

1. Cost of postage

2. Protectionist import taxes. Here is Europeans list of things they are thinking of levvying punative taxes on. It's long. http://trade.ec.europa.eu/tdi/notices.cfm

3. Regulatory approval. The burden of bringing a new electronic device to market is pretty high. You need specialist facilities (or to pay a lot of money to a test lab) to prove electrical emissions are fine. This is why so many Chinese designs never make it to the UK. Alibaba is full of lovely android powered arm net-tops, but you note the ones you see in electronic stores are from the same limited range, even the cheap ones. This is why.

I presume the same is true for pans. So although designing a pan and bringing it to manufacture is simpler than manufacturing a new ASIC or even embedded computer, only a very few pan designs can be sold in the UK. It just isn't worth bringing another one through approval (unless it's much cheaper). And the approved designs can sell for much more.

One hears about western companies keeping as much of the 'value add' as possible out of China and India, possibly because it's so expensive to retrieve money once it's in there. I wonder if this also is part of it.


"So, why do pans get so much more expensive as they cross the ocean?"

My guess? Because pans get sold by experienced companies who recognize that the price of an item is decided by its utility and the amount people are prepared to pay for that utility, not anything to do with the cost of manufacture. The gap between the manufacturing cost and the utility value of an item only lets you choose whether or not to be in the business of selling that item.


In Poland such pans generally cost less then 10 USD (and they are made in China, probably in the same factory). So yep, it seems the prices of a pan are mostly determined by customers purchasing power.

Transport costs per unit are probably also higher for pans (cause they're bigger and heavier than arduino).


>Because pans get sold by experienced companies who recognize that the price of an item is decided by its utility and the amount people are prepared to pay for that utility, not anything to do with the cost of manufacture.

The price of an item is decided by supply and demand. The only way you get a retail price which is significantly higher than the manufacturing cost is a lack of competition.

For items like pans this is probably more at the retail level than the manufacturers. I kind of doubt Walmart is buying $1 pans for $45 and selling them for $50.


No, I think this is where the "pan selling industry" and the "tech toy/trinket selling industry" are different.

There's something going on in the "pan selling industry" which results in very little "race for the bottom price" type competition - there are clearly many individual companies importing pans from China, but the "well known retail price" of those pans is ~$30 no matter which retailer you buy it from (within reason - at least within enough reason that all the $30 price-point retailers are still in business).

Tech toys, on the other hand, quite often end up in the "race to the bottom" price category - with many innovative and desirable items ending up available from sites like Alibaba or DealExtreme at ludicrously low prices.

I think part of it is the audience - the "pan buying demographic" is probably a lot earlier in it's uptake of globalised web based purchasing than the demographic who's buying Arduinos and RasberryPis. Perhaps one day your average mid-western housewife will say "I need a new saucepan, I'll just check Alibaba before I drive to the mall and pick one off the shelf at Walmart - if I can save $20 and only need to wait a week I'll just get it delivered."

But like I said - I suspect it's "experienced companies" who are not exactly colluding or price-fixing, but who all know that it's better for _all_ of them to keep pans priced at ~$30 and have everybody make margins of almost $29 per sale, rather than end up trying to compete on price at $5/pan knowing that the total market for pans won't increase just 'cause the prices drop 85%.

(And sometimes, you get the evil-Walmart-effect, where someone big enough to absorb the temporary losses chooses to drop the consumer-expected pricing down to $2/pan, waits out until the competition goes broke and vanishes, then puts the prices back up near $30/pan while owning all or most of the market.)


>There's something going on in the "pan selling industry" which results in very little "race for the bottom price" type competition - there are clearly many individual companies importing pans from China, but the "well known retail price" of those pans is ~$30 no matter which retailer you buy it from (within reason - at least within enough reason that all the $30 price-point retailers are still in business).

That's what I'm saying -- they don't compete with each other. There isn't enough competition that anyone decides to try to improve their market share by lowering prices, because it's just Bed, Bath & Beyond and Sears and such like "competing" with each other in any given local area, and not enough people buy housewares over the internet to move the needle against those guys.

>I think part of it is the audience - the "pan buying demographic" is probably a lot earlier in it's uptake of globalised web based purchasing than the demographic who's buying Arduinos and RasberryPis.

More than that, electronics are purchased in bulk by corporate purchasing departments. If you're buying 25 new devices every two years you may not be such a big fish that you qualify for a volume discount, but you're certainly spending enough money to justify a significant amount of comparison shopping, which means a high price elasticity of demand and an immediate market share advantage for retailers who engage in price competition.

>But like I said - I suspect it's "experienced companies" who are not exactly colluding or price-fixing, but who all know that it's better for _all_ of them to keep pans priced at ~$30 and have everybody make margins of almost $29 per sale, rather than end up trying to compete on price at $5/pan knowing that the total market for pans won't increase just 'cause the prices drop 85%.

The phrase you're looking for is "conscious parallelism." You get de facto collusion without communication by means of everyone adopting the strategy that they not be the first to engage in price competition. It works only so long as you have sufficiently few competitors that all of them religiously follow the strategy -- because having a third of the market with 5X the margins is more profitable than having the whole market at 1/5th the margins. The problem comes when you have more competitors, in which case someone decides they would rather have half of the market at 1/5th the margins than 1/15th of the market at 5X the margins, and believes (perhaps correctly) that they will be the one to survive a price war. This has basically been Walmart's business model for probably the majority of the products they sell, and Amazon et al are now doing the same thing even more aggressively on the internet.


You're assuming a free market, which stops working when you have too much differentiation between products. Free market analysis stops working when you consider that at one point humans are part of the decision making process


You can buy those same cheap frying pans in restaurant supply stores without a lot of markup. They're shitty pans (very light) and restaurants go through a lot of them.

Edit: Here's an example, but at your local store in a sketchy area they'll be even cheaper: http://www.instawares.com/winco-gladiator-fry-pan.wdi-afp7xc...


This is why as an intern at NASA I decided government work wasn't for me and I eventually ended in startup world. However, I think there is something to be said for creating within strict guidelines and procedures, it can sometimes lead to amazing engineering achievements.


This also shows in how quick you can iterate. The "REPL time" in mechanical engineering has decreased a lot with CAD software but it's still insanely long compared to software development.

I used to work in a lab with a researcher who designed complex mechanical systems with lots of rotating parts made from rare and expensive materials. Everytime he needed even the slightest modification he had to wait for at least three months to have a new part machined.


This happens all the time in government. Business has the lucky position of constantly scrambling to cut costs, cut redundancy, cut inefficiency. Otherwise they'll be beaten by a new company that isn't tied down by self-induced paralysis. The government has no such incentive to do so. When faced with a choice between saving fifty billion dollars and having the assurance that everything is hunky-dory, the government will spend the fifty billion dollars without a second thought.

One of my duties, along with being a radio technician, is ordering replacement parts for the systems that we work on. Some of the stuff is expensive and reasonably so - for example, a circuit board with $200 bucks in components is $20,000 because it has to be professionally made and is basically a one-of-a-kind part. The supplier has to charge that just to recoup the costs of retooling. We accept this because we need very reliable parts on radar systems and are willing to pay for the assurance that these parts will be good.

On the other hand, I just ordered an audio cable through the FAA. Simple two-wire audio cable... $333.45. I could've bought the exact same thing from Radioshack for five bucks. Or Monoprice for one dollar.

At one point, Raytheon told us that we were not allowed to do any intermediate-level maintenance (anything involving fixing circuit cards) on their stuff. We were expected to send the bad cards to them, and they would give us replacements and bill us for $10,000 each. The chief warrant officer laughed, said "Fuuuck no," and told us that if we could easily fix it, do so.

Raytheon realized we weren't sending them any circuit cards and called the commanding general to get him relieved. He didn't get fired, but we started sending the cards to them for broken 2-cent resistors. For ten grand each. Same thing with $30,000 power supplies, etc.

Right now, the government has declared that getting office supplies from Staples is horrible. We're supposed to get them from Servmart, which sells them for five times the price... or more. Cheap-shit ballpoint pens for a dollar each when I can get Bics for $4.60 for 72 of them.

Your tax dollars at work, gents.


How a comment cost $10,000 and a car cost $5: I got you all to read this, wasting your time and electricity, computer costs, etc. In contrast, I got a cab yesterday and it only cost $5.

I don't generally insist that everything posted be transcendent, but I fail to see even a fair point being made in the post. If the point is "buying things for an aerospace corporation is more expensive than renting an hour on EC2", okay, but we sort of knew that already.


I work in hardware, and wanted to give my perspective on this.

Designing and building hardware is expensive, really expensive. If you don't take the time to follow proper procedures you are not only wasting your time, but often hundreds to thousands of dollars of parts you are never going to use. Going ahead without thinking can easily put you 2 or more weeks behind just because that is how long it takes your supplier to make what you ordered. You don't just quickly spin up an instance of a test, you have to buy or rent equipment that is often in the thousands to hundreds of thousands of dollars range.

I can see exactly why you wouldn't want to just go to Home Depot when working on equipment like what MDA makes. Say you go to Home depot and buy your fasteners, now the company has to pay for your time and reimburse the cost, which is almost certainly going to be as much or more to process than the courier charge ($30-$40) to ship the parts from the company you are making monthly payments to regardless. Then there is the additional risk of bringing in parts that could get mixed up with real production parts. There is a huge risk if you somehow get that Home Depot fastener mixed up with the special aerospace fastener you are using, the extra cost is worth it as insurance that "bad" fasteners aren't getting into the product. I can say from experience, it is ridiculously easy to get parts mixed up even when you are careful.


I fail to see how filling out $2000 in paperwork will stop you from getting screws mixed up. Because that's what the cost was, apparently. Not the actual equipment.


Those rules are important. Not every company in the world is a start up with <100 people. Once you have thousands of employees, you need controls. You can't have interns selecting suppliers or putting stuff into the bill of materials, are you serious with this?


Unfortunately we're just as much to blame for the red tape. It didn't get created out of a vacuum, it was created to protect people. I wrote a whole blog post called Why Does a Hammer Costs $5000 at http://www.followsteph.com/2011/11/22/why-does-a-hammer-cost...


why, in Russia, on government sponsored Olympic village construction projects $2k for a few screws is strikingly cheap.)




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: