
Coding Error Forces 2019 Subaru Ascents to be Scrapped - georgecmu
https://spectrum.ieee.org/riskfactor/computing/it/coding-error-leads-293-subaru-ascents-to-the-car-crusher
======
userbinator
_Subaru indicated in the recall that “there is no physical remedy available;
therefore, any vehicles found with missing welds will be destroyed.”_

Something doesn't feel right, when the cost of adding the missing welds is
somehow higher than the cost of a whole new vehicle...

If missing welds were discovered on an airliner, for example, the fix would
almost certainly be to add them, and not replace the whole plane.

~~~
grecy
A fun anecdote to explain the absurdity that is Auto manufacture.

When Ford made it's own vehicles in Australia, the engines were made in
Melbourne, while the body and final assembly of the two was in Geelong. That's
an hour away on the busiest freeway in the country

A friend is a process Engineer and did a big investigation into what they
could do when the engines were late arriving due to traffic jams. Of course
the whole thing was Just In Time because reasons, so they needed the engines
to arrive just at the right time.

At the end of a multi-year study it was concluded the most cost-effective
thing to do when the engines were not ready to put into cars that were ready
for them, was..... let the cars fall off the end of the line, crush them, and
eventually crush the engines when they showed up.

Any attempt to re-integrate them, or salvage stuff, or push them back on the
start of the line, or anything else you want to think of costs more money than
the above solution. They ran the numbers every which way, and as shocked as
they were, it's the truth.

It was a rumor for a long time that every single time the production line was
stopped, even for five seconds, it cost $1Mil. It was investigated, and found
to be higher than that.

~~~
sitkack
Sounds like having the plants separated by a road with traffic jams was very
expensive.

Probably way more expensive than colocating the two factories.

JITs are really susceptible to shock.

------
martyvis
I still remember when I started playing around with computers, 40 years ago in
a Tandy store on TRS-80, you were told something like "it doesn't matter what
you do on the computer, you can't hurt it or damage anything". Now everyday
when I work (on networking and security for large enterprises) I constantly am
thinking about how I have to really make sure the process I am about to
execute isn't going to break something or disrupt someone's business
operation. Rarely do I have to contemplate that what I get wrong might kill
someone though!

~~~
9wzYQbTYsAIc
Here’s an interesting series of articles on Risks to the Public by the ACM
Special Interest Group for Software Engineering:
[https://dl.acm.org/results.cfm?query=Risks+to+the+public&Go....](https://dl.acm.org/results.cfm?query=Risks+to+the+public&Go.x=22&Go.y=4)

------
rustcharm
This was really an inspection error. A human can miss a weld, too, so
manufacturing lines rely on inspections. Someone was signing off on work that
wasn't done. Whether a human or a robot was supposed to do that work is a nit.

~~~
9wzYQbTYsAIc
From TFA: “[Subaru] found an Ascent with the missing welds during a routine
sampling inspection on 21 July”

The affected vehicles were built between 13th and 21st.

Not sure how long QA takes for a vehicle, but they found it within a week.

~~~
Timmah
_sampling_ inspection. Not a hard software check: `assert ( numWelds == 30);`

Or a camera that takes a picture of each vehicles undercarriage. They probably
thought that was unnecessary until now.

~~~
9wzYQbTYsAIc
Could have been an error of requirements causing both the robot control code
and any automated testing to be incorrectly implemented.

------
pfarnsworth
I'm interested why over-automation caused so many problems in manufacturing
plants.

I would think that once an assembly line has been set, full automation would
be completely ideal and perfect. Is it because plants are always changing and
getting tweaked, and changes require software updates, which invariably have
bugs and take time to adapt to changes?

~~~
trentmb
> I'm interested why over-automation caused so many problems in manufacturing
> plants.

A robot will uncritically do what you tell it.

An experienced welder will tell you you're an idiot when you're an idiot.

~~~
Someone1234
So is the issue over-automation or bad QA after the system is set up?

~~~
setr
I would imagine theyd try to automate QA too, akin to unit testing

You could imagine a situation where the automated QA is trusted and does its
job in sets, with humans interspersed to backup the QA programs. But one of
the QA checks is actually buggy and OK’s the missing weld, but the full batch
of work covers it up before reaching a human for the double-check

In which case, both issues are at fault: too much reliance on the QA program,
and allowing too much work to be done before it reached a human (bad double-
check setup)

~~~
archi42
I imagine if you have a list of welding points and feed that list to both the
assembly and inspection robots, stuff like this happens.

Especially in case the robots share code and e.g. have the same parse error
when loading your list of welding points.

------
logicandreason
Coding error? Maybe... but I would bet it all started in business where some
management decision was made to replace "Bob" who programmed the bots for
years was replaced by a cheaper less experienced person, all in efforts to
reduce IT costs. So let's rename this thread, "Business Management decision
forces 2019 Subaru Ascents to be Scrapped".

------
paulcole
Despite being the title of the article it isn’t what it seems. The coding
error was in the production of the cars which led to them needing to be
destroyed.

The coding error didn’t literally send them to the crusher instead of the
showroom floor as the title implies.

~~~
seattle_spring
I was picturing some oversized Subaru-branded Rube Goldberg machine, with the
final step being a shiny car being tossed into a crusher.

~~~
paulcole
Now that’s news!

------
stevev
When you move your plants to the US.

