Hacker News new | comments | show | ask | jobs | submit login
The $8M Replacement for a $20 Dollar Fan (2013) (stlcc.edu)
75 points by kilroy123 on Sept 30, 2017 | hide | past | web | favorite | 60 comments

This is an urban legend or folk tale or whatever you want to call it. I've seen this exact same story before only it was a soap factory in Japan or something like that. It's presents a good lesson but isn't a true story at least this re telling of it isn't.

Like many of the stories on The Daily WTF, I consider this to be in the genre of stories that are impossible to determine if they're literally exactly true as told, because there are so many things like that that happen all the time, so by the time you factor in just a bit of distortion it matches too many things.

If it isn't literally true, and I have no idea, many many things just like it are.

Heck, I could probably collect a few dozen stories from HN folk right here about projects they've done where they've wasted tens of thousands of dollars (and up to arbitrary amounts of money) delivering something that their customer didn't want and anybody could have found that out with a single question in 10 seconds, if anybody had thought to ask OR if management had taken the answer seriously.

If we’re swimming in relevant anecdotes, why is it always this same apocryphal crap version which is always spread? Maybe the difference between the stories you would collect and this story, is that the stories you would collect would have more than one side to them...

"why is it always this same apocryphal crap version which is always spread?"

It's the winner of the memetic evolutionary war. No sarcasm. It doesn't mean much. I think that you're likely correct that the more complicated and nuanced versions don't survive, because they transmit less well. It's still not exactly a stretch to say that lots of money gets wasted all the time for very, very similar reasons.

For most of the anecdotes I have, they are all covered under a NDA. This one is a good one to repeat as I don't have to worry about someone complaining that I gave away their trade secrets.

Most of them leave the $20 solution in place as they can then save $$$ over paying for a 'proper' solution (someone typically comes up with the $20 solution before the $8m solution makes it to the RFQ stage).

Because the apocryphal crap version makes a nice story. It's like the space pen story, or the radio exchange between a U.S. battleship and a Canadian lighthouse.

I feel there is a more recent upgrade to this type of story where Elon Musk ramps up production of one of the early Tesla vehicles by talking to factory workers and swapping out lines or parts about 10x+ faster.

It's a parable.

There's a similar anecdote out there, also involving toothpaste. In this one the toothpaste company spends millions on a new marketing campaign to sell more toothpaste and then someone suggests that they make the hole in the tube bigger so people use more.

A real example of this would be Alka-Seltzer(dissolvable antacid/pain medicine). An ad showed two tablets in the glass, and they saw a large increase in sales. Nowadays they simply come in packets of two.

https://en.wikipedia.org/wiki/Alka-Seltzer#Marketing http://www.snopes.com/business/genius/alka-seltzer.asp

Ever noticed how tooth paste ads always show the entire brush covered in paste, even though a lil' dib is already enough?

Freshy Fresh+ for YOUR health

there's a similar one too about NASA researching a pen that works in space.

At least that’s a real pen, albeit not developed by NASA.

The lesson of cost vs. benefit is what's important here, IMHO.

Apparently it needed the $8M replacement for a $20 webserver

The maintenance guy already spent $20 and fixed the issue. Why spend $20 more to fix something that isn't broken? You'll take on webserver maintenance with that, so a webserver doesn't really cost $20.

Anytime you want to replace a simple analog solution with a complex digital solution, your costs aren't 1to1.

I think the point the parent was making was that the website was down. ;)

HA! Thank you. That had gone right over my head :)

What jtbayly said.

Also, now I can actually read the article!

See, if they would have spent $10Mil and also had a swivel arm that pushes the box off and restarts the machine, Bart would never have had to selfishly innovate and you'd have the architecture at most companies.

See also, the space pen:


Which in fact did happen and is (in my opinion) a far more interesting story in this vein.

I used to work with a consultant who was a bit of a maverick and would regularly get dismissed from his clients after he ignored all the top brass and immediately went to talk to the people on the floor to find out what was really going on. Nearly every time his radical solutions and reports, which made sense, would just get filed in a drawer because they got rid of middle management. He didn't care if they implemented his ideas. He'd just move on to the next.

In "Surly you're joking my Feynman" there is the story of RF talking to the painter who teaches him to mix paints, and the colours you need to get make yellow. Turns out you should talk to the person who's been mixing paints all their working life.

Isn't there a similar story after the challenger accident where he talks to the maintenance people on the ground?

Shouldn't the title be the other way around... $20 Fan replaces a $8M process

"The service is unavailable."

Also archive fo https://archive.fo/05CkQ https://archive.fo/YtM2b edit sorry not working might need to archive it again later

The fan doesn't ensure quality. It is possible for an empty tube not to get blown off.

But then the 8mil solution would catch it.

Exactly but it takes the bite out of this anecdote, which is that the $8m solution is unnecessary because there is a $20 solution.

Let me guess, it was hard to get to it

The story is obviously intended to teach a lesson and not a real piece of truth. But the lesson is real, and we’ve all seen it in action.

Product people want to push products. Management wants to manage. Engineers want to engineer. Nothing about any of this is inherently bad, but putting these three motivations together in a vacuum can have this effect.

This is all, in my opinion, an example of resume driven development. Everyone wants to build themselves up. In a competitive market, you have to. For product people that means delivering products, for management people that means managing problems and finding successful solutions, for engineers that means engineering cool shit that works.

We are strongly motivated to behave this way.

The point of the parable is that these motivations don’t necessarily lead to the optimal solutions. And that the difference between the highly productized, well-managed, elegantly engineered solution and the quick fix is often not large.

But there’s a strong reaction against boring solutions. Boring solutions don’t get you the next gig. Boring solutions don’t impress anyone. Boring solutions don’t make your resume pop.

This is a grossly exaggerated example, of course, but that’s how parables work. I’m kind of surprised by how many of the comments are missing the point.

This tendency to over-engineer, over-productize, and over-manage is everywhere in the tech industry. We do it everywhere. Not to start a flame war, but I’m Looking at you, new JavaScript framework flavor of the month. I’m looking at you, infrastructure platforms and noSQL solutions.

The thing that we don’t seem to be able to accept is that the vast majority of the problems we solve day in and day out at our startup companies and our big big companies are very straightforward and boring problems, for which there already exists a boring solution.

This is more specifically true of web development than native dev work or embedded systems, and there are obvious places where this is entirely untrue. Cutting edge research in theoretical comp sci, embedded systems, AI, and cryptography.

And I say this as a web developer, not one of the other interesting things I mentioned above: most of what we do is and should be boring. They are simple interfaces to data. A very few of them—-maybe .01%—have to actually deal with scale.

I know I sound like a grumpy old man, but that’s increasingly what I feel like.

One of my most recent projects reflects this so closely that it’s not even funny. We had a project without enough time, we wanted to bring in a vendor to deliver a system-wide platform. Vendor couldn’t deliver on time, so we had to do something else.

My solution took a day to prototype and another few days to deploy. It was boring as fuck. No one would be even remotely impressed with the implementation. It was a boring problem with a boring solution. But it solved the problem, and we were free to move on to something else.

If we want parables like this to go away and stop offending us because they aren’t literally true, we (especially us web developers) need to stop pretending that we are so special. Product and Management should also do the same thing.

Most problems are boring, and they deserve boring solutions. Yeah, I know that doesn’t help your resume or mine, but you don’t get paid by a company to bump your resume. You and I get paid to solve mostly boring problems. Until we are willing to admit that to ourselves and each other, this parable is going to remain relevant.

The core of the article (See full version for the extra commentary):

A toothpaste factory had a problem. They sometimes shipped empty toothpaste boxes without the tube inside. This challenged their perceived quality with the buyers and distributors. Understanding how important the relationship with them was, the CEO of the company assembled his top people. They decided to hire an external engineering company to solve their empty boxes problem. The project followed the usual process: budget and project sponsor allocated, RFP, and third-parties selected. Six months (and $8 million) later they had a fantastic solution – on time, on budget, and high quality. Everyone in the project was pleased.

They solved the problem by using a high-tech precision scale that would sound a bell and flash lights whenever a toothpaste box weighed less than it should. The line would stop, someone would walk over, remove the defective box, and then press another button to re-start the line. As a result of the new package monitoring process, no empty boxes were being shipped out of the factory.

With no more customer complaints, the CEO felt the $8 million was well spent. He then reviewed the line statistics report and discovered the number of empty boxes picked up by the scale in the first week was consistent with projections, however, the next three weeks were zero! The estimated rate should have been at least a dozen boxes a day. He had the engineers check the equipment; they verified the report as accurate.

Puzzled, the CEO traveled down to the factory, viewed the part of the line where the precision scale was installed, and observed just ahead of the new $8 million dollar solution sat a $20 desk fan blowing the empty boxes off the belt and into a bin. He asked the line supervisor what that was about.

“Oh, that,” the supervisor replied, “Bert, the kid from maintenance, put it there because he was tired of walking over to restart the line every time the bell rang.”

Another way of looking at this story:

It took $8 million to take a problem that was only visible to the sales department -- empty boxes are hurting our image! -- and make it visible to the stakeholders who had the right insight to actually fix it -- the people on the production line.

That is it cost $8 million to move the pain point to the "right" spot.

Even if this story was true, neither solution is good as they still have wasted, empty boxes. The solution would be to figure out how the empty ones are produced and fix that (root cause analysis) The scale would be part of the solution to verify the fix (continuous integration)

> The solution would be to figure out how the empty ones are produced and fix that (root cause analysis)

Maybe.. maybe not. One possible outcome:

Likely problem: speed of production. Solution: slow down production. Analysis: the loss on sometimes empty boxes is much lower than the loss on a slower production line. Conclusion: the fan is the correct solution.

That line of reasoning is what cost them 8M in the first place. If it looks stupid, but works, it ain't stupid.

I dislike this line because life is more nuanced than it portends.

>>I dislike this line because life is more nuanced than it portends.

Most of the time, it isn't. That's the point.

Exactly. Hacker news

The solution is good, but it's not perfect.. it's just a question of how much time and effort particular problems are worth.

In fact the whole lesson here is about exactly this.

Thanks for the excellent TL;DR.

Now if you can turn yourself into a summarization bot....

It's not a summary, just cutting out the commentary.

If only there was a word for taking a long article and shortening it, while preserving the main points.

"Here is chapter 2, it's the only chapter you need" is not a summary of a book.

A summary is when you pull the relevant points out of the entire document.

Especially when the part quoted here was a quote inside the original article. The entire native text of the article was discarded.

$8M still seems high for a scale.

That's because the story is bogus. Weighing products for QA (to make sure there wasn't underfilling) is extremely common. It's called "checkweighing" and you can buy COTS scales to do exactly that.


100 000 for the scale, 7 900 000 for knowing where to put the scale.

100,000 still seems high. Clearly I'm in the wrong industry. I should be fleecing enterprise businesses instead.

Let me introduce you to a thing called Java!

Yeah, a lot of the lesson seems to (unintentionally) be "don't outsource small things". Even completely ignoring the fan solution, the CEO could have assigned one existing engineering employee to work on it and gotten an $8k system with a non-precision scale.

Yes, but $8MM is total project cost = existing full time employee costs, outside consultant cost, and hardware + software.

This bit of apocrypha is older than me, and never improves. At this point it’s almost part of a the almost religious ideology around some Libertarians.

Yeah lots of variants of it too. I've heard it told in terms of nationalities too "In nation X they built a multi-million dollar sensor, in nation Y they just used a fan." In that format it's similar to the NASA space pens.

The real space pen story is so much more interesting too! Of course the last thing you want in a spacecraft is graphite dust literally “pen testing” every system.

The $8M Replacement for a 20 Dollar Dollar Fan

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact