
How a lawyer, mechanic, and engineer blew open an auto scandal - waffle_ss
http://pando.com/2014/10/18/gms-hit-and-run-how-a-lawyer-mechanic-and-engineer-blew-the-lid-off-the-worst-auto-scandal-in-history/
======
vermontdevil
This article illustrates the need for trial lawyers and keeping that route
option open.

See how Texas shut down the medical lawsuits under the guide of malpractice
reform where it didn't really work as advertised.

In an ideal world, the consumer protection agencies, etc would do their jobs.
But they can't because of the revolving door syndrome, the industry lobbying,
and so on.

So what are our alternatives to force companies to fix their product defects?
People actually died due to their shoddy design/production and these companies
tried to hide or cover up.

I wouldn't be surprised to see an increased effort to pass laws to remove the
ability for us to sue companies in case something goes wrong. This on top of
the media campaign to paint trial lawyers as the scum of earth. Some may but
many are actually very good and very dedicated at what they do.

~~~
rayiner
It's probably self-serving, but I think it's valuable to have people in
society who have different incentive structures. Trial lawyers don't answer to
share holders or analysts--they answer only to judges and juries composed of
ordinary people off the street. Within an otherwise capitalistic market
economy, having some of them around provides a valuable foil.

Tort reform in Texas worked out as designed--what you have to ask is: what was
it designed to do? Take caps on jury damages: do you think that's targeted at
frivolous litigation? Or is it targeted at limiting compensation in the most
meritorious cases, ones where someone really fucked up and a jury saw fit to
award a big number?

------
lovelearning
I feel the root cause here is the culture of setting hard deadlines by
management.

Getting a deadline moved even for logical reasons involves pushing against
massive internal red tape. Missing a deadline invites greater punishment from
management than compromising on quality. Naturally, quality loses out.

Happens all the time in our own industry, and sadly, this case shows that it
happens even in safety critical industries.

All engineers, regardless of industry, should be trained in how to negotiate
with management. All managers, regardless of industry, should be trained to
respect opinions of subordinates.

~~~
amirmc
That almost sounds like you're absolving the engineers (as though they can do
no wrong). Changing a part, without also changing it's part-number, doesn't
sound like a normal thing to do (and then forget about).

This is a complex issue and I do not agree with the (over-simplistic)
reduction to 'management' and 'deadlines'.

~~~
lovelearning
Not absolving the engineers. The engineer in this case - and other cases - may
well be guilty - both legally and morally - of mistakes.

What I'm trying to understand is, what would make an experienced senior
engineer prefer such shortcuts over doing the right thing.

The article says, "Faced with a deadline, DeGiorgio replied: If increasing the
torque will destroy the switch then do nothing. Maintain present course. Under
no circumstances do we want to compromise the electrical performance of the
switch.".

I interpret it as the deadline being a root cause. At the heart of any complex
issue lies one or more root cause(s). What were they here? What made him
behave that way? Fudging the records is a consequence, not a root cause.

~~~
fnordfnordfnord
>What I'm trying to understand is, what would make an experienced senior
engineer prefer such shortcuts over doing the right thing.

Organizational failure. An onerous documentation change/approval management
scheme. Covering up a mistake. A culture that causes people to fear losing
their jobs, especially for cost reasons, or the feeling that company
management are looking for reasons to cut staff. Having your group be blamed
for an expensive product recall makes you an easy target. One of the last
things a middle-aged engineer in the US auto industry wants to be right now is
laid-off.

We naively expect a senior engineer to do the ethically correct thing without
respect to whether it has negative personal consequences. The problem with
this idea is that the consequences for being "that guy" are too great; and
both engineering societies and state licensing boards have been derelict in
their roles of helping to protect engineers who refuse to do the ethically
gray/wrong.

>I interpret it as the deadline being a root cause. At the heart of any
complex issue lies one or more root cause(s).

There are so many things wrong with product engineering management, but yeah
laying a deadline that no engineer has the courage to break would definitely
be one of them.

~~~
zaroth
It doesn't sound like such a horrible position to be in; working for a multi-
billion dollar company that was trying to force you to ship a part that could
kill people. Whistleblowers get 1/3rd of damages, it's like wining the lottery
for ethical people.

How long does it normally take to design one of these switches, when you've
been doing it most of your life? Knowing absolutely nothing about the
complexity of these switches, but knowing what small hardware teams have been
able to accomplish at companies I've worked at, 3 years sounds like a more
than reasonable time frame.

~~~
fnordfnordfnord
>It doesn't sound like such a horrible position to be in; working for a multi-
billion dollar company that was trying to force you to ship a part that could
kill people.

It's probably not a case of some moustache twisting monocle wearing fiend who
was the manager and the engineer was absolutely convinced that it was a faulty
part that would ultimately be responsible for death and destruction. It was
more likely one of a number of similar sub-assemblies, all having some
questionable attributes and people trying to ship things under the guise of
"perfect is the enemy of the good". At some points along this road, various
people probably began realizing that there was a real problem, one that they
might have some culpability in. Why the people involved continued to make poor
decisions, I don't know.

>Whistleblowers get 1/3rd of damages, it's like wining the lottery for ethical
people.

It's a nice theory. I've yet to see it happen consistently in practice (or
pretty much ever). Engineering ethics case studies are littered with cases
where engineers tried to alert management of problems and were punished for
their efforts.

>How long does it normally take to design one of these switches, when you've
been doing it most of your life?

Not very long given ample resources and nothing else to do. I doubt they were
working under those conditions.

------
derekp7
Question: What were they trying to achieve with the 2005 version of the
ignition switch, which required a new design, that couldn't be provided from a
1975 switch design? It seems that a lot of effort and money is poured into
redesigning every component of products every few years, when this was a
solved problem many decades ago. After all, the ignition switch has to 1) be
turned with an authorized key, 2) provide electrical contact for accessory,
run, and start positions.

~~~
Thimothy
Have you had a car made in the last 15 or so years?

Nowadays ignition switches do everything but starting the ignition system. As
the car is fully controlled by a computer, the key is an electronic gizmo that
authenticates the driver as the owner and signals the starting sequence. The
revolving movement is completely unnecessary and just provided for familiarity
purposes, there are cars that instead of a key have a card and start by
pushing a button.

My bet is that maybe in 2005 GM was building the first switches under this
principle, thus the need to preserve the electrical characteristics of the
system, so the data transmision channels between the key and the computer
wouldn't get compromised.

~~~
derekp7
Ah, that explains it -- my F150 pickup truck is 14 years old, the only
technology it has is an RFID chip in the key (which I assume is read by a
sensor in the location of the ignition switch, fed to the engine computer, but
otherwise not requiring any other re-design of the key switch).

------
ploxiln
When I read this story, I thought of parallels to large complex software
projects (of course). You could call a large software system that would take
more effort fix than has been been spent to create it so far "totalled", like
a badly crashed car. It happens (usually with government payroll systems it
seems).

GM seems to have "totalled" their entire lineup of cars designed and produced
over the last 20 years - the quality of their organization's work has been so
low that it's not really possible for GM to find and fix all the serious
problems on all the cars they've made. They would have to dissect all the
parts from a sampling of cars of each year, since they can't even trust that
the same part number is really the same in different years! It's probably
cheaper to make a couple new car models, really make sure they're not
defective, and provide everyone who still has a GM car with a replacement car.
Cheaper, but maybe still not possible. "Good enough" massive recalls will have
to do.

------
gregpilling
So the writer describes the guy who does the CSI on the car as a "grease
monkey" and implies he is an uneducated hick. Yet it was his work that figured
it all out.

Have you ever used a Tech 2 scanner to diagnose one of those cars? They are
quite complex, with mutliple computers networked over the canbus system.

Grease Monkey. Like Code Monkey or Porch Monkey. Derogatory.

~~~
exogen
I didn't read anything implying he is an uneducated hick unless you think the
term "grease monkey" itself has that implication. But I don't that that term
has the connotation you're saying it does. Whenever I've heard "grease monkey"
it seems more like the car equivalent of "hacker" – that tinkering with cars
is something you deeply understand and enjoy, not just a job.

~~~
gregpilling
"Charlie Miller has been a grease monkey for almost his entire life. Born in
Tippo, a tiny town in the heart of the Mississippi Delta, Miller overhauled
his first engine when he was 12. He put himself through a local college
working as a mechanic while supporting an addiction to drag racing and after
he graduated he become a professional hot rod driver. In 1974 he opened an
auto repair shop in Merigold, but it was 15 years before he was asked to offer
an expert opinion, and only because of happenstance"

That didn't sound very complimentary to me. Maybe it could have been written
like:

"Charlie Miller was asked by a friend to help in a court case. His friend has
asked for help because of Charlie's unique background. Charlie had a love for
cars since he was a child, and was smart enough to overhaul an engine at the
age of 12. He was not only good with tuning an engine, he could also drive. He
competed in drag racing as an amateur before becoming a professional driver. A
graduate from a local college, Charlie was able to combine his love of cars,
his self directed education and his keen diagnostic sense to solve his friends
problem as an expert witness, and that is how he started his path to become
the guy who solved the biggest recall in automotive history. "

~~~
PhasmaFelis
There is nothing insulting in the bit you quoted. It describes someone from a
rural background who turned a teenage racing hobby into not one but two
careers. Maybe you're so sensitive to country folk getting shit that you think
just mentioning his rural background must be done with a snicker and a sneer,
but I don't. It's the plain facts.

And "grease monkey" is not an insult. It's more of a compliment, if anything.

~~~
gregpilling
If it is such a compliment, then why not go to an auto shop and say "Hey you
grease monkeys, how are you?" and then see if you get punched. I am not brave
enough to do that, and I have been working in automotive for 25 years.

It would not be the same as talking to computer programmers and calling them
'hackers'

------
scotty79
Absolutely horrified.

> The Cobalt’s engine shut off and the lights inside and outside the car
> dimmed. Melton hit the brakes, but no power from the engine meant no anti-
> lock brakes and no power steering.

I wasn't aware that anti-lock brakes (is that ABS?) is not a purely mechanical
system. Also, was power steering hydraulic or electric? Shouldn't it be run of
battery for short period of time?

> While Brooke’s lap belt glued her waist to the seat, her shoulder harness
> went slack the instant the engine shut off.

Now that's just criminally stupid.

EDIT: I guess I need to read through to the end to the article but that's a
hard read for me for some reason.

~~~
Animats
That's not quite right. The Cobalt, like a lot of cars, has a seat-belt pre-
tensioner. This is an explosive device that pulls in the seat belt by about 4
inches when the air bags fire. The seat belt's basic inertial locking function
still works without power.

Anti-lock brakes, on the other hand, definitely require power. They're
computer-controlled, using pulse counters in the wheels, electrically-
controlled brake valves, and rate gyros. With ABS, jamming on the brakes is
just fine. Without ABS, jamming on the brakes is awful. In the first crash
mentioned, the driver not only jammed on the brakes, but having entered the
skid, turned away from the skid, which makes it worse. It was also a side
crash, so the air bags would not have fired. Who practices non-ABS skid
recoveries in cars any more?

Losing electrical power in that car is a big deal. You lose propulsion, power
steering, power brakes, anti-skid braking, seat belt pre-tensioning, and air
bags all at once. That's bad design. All those electrical functions should be
enabled until the transmission is in PARK.

~~~
x0x0
And the airbags wouldn't go off, either, without the key turned on.

But having your car suddenly go from having abs -- where jamming the pedal
down is probably the right thing to do -- to a state where the driver must
pump the pedal is probably more than virtually all drivers can handle.

~~~
emp_zealoth
Given my experiences i personally feel she has shown terrible lack of skill -
why slam on the brakes? The car was fully controllable. I`ve had my engine
stall once and i mistakenly turned my engine off (and the hydraulic power
steering went off) midturn and it wasn't a light car. I was completely fine.

On the other hand i find american buisness practices despicable and sickening.

~~~
msl
> why slam on the brakes?

Maybe she wanted to stop the car she had lost control of?

The reason for ABS to exist is that slamming the brakes is a very natural
thing for (most of) us to do in a dangerous situation. If you're designing a
car for a human to drive, you should keep this in mind.

~~~
rodgerd
To amplify that observation: ABS was banned from Formula 1 because it made
braking too easy. Modern ABS can outperform people who are paid in excess of
$40 million/year to be the best drivers in the world, who have unreal
reflexes, complete mastery of a vehicle, and the ability to make good
decisions under high-pressure driving situations.

The idea the average driver is an adequate substitute for ABS is laughable.

------
segmondy
For those of us who are programmers, if you wrote software that lives depended
on, how confident will you be that lives won't be lost? My take away from this
is to work harder towards excellent, better quality control, more
documentation and don't change a working system without following process that
informs other.

In the software domain, there's this bravado about hotfixing an issue in
production. Raymond DeGiorgio did that, he fixed the problem without going
through proper channels, not thinking of "legacy" cars, and those one's
failing to get patched, cost lives.

------
concernedhner
At some level, I am horrified at what was done with prior knowledge by this
institution. At another, this incident makes me wonder how this ties into the
diffused nature of responsibility within corporations. Was the diffusion of
responsibility within these corporations was so great that they could
perpetuate these acts without any one individual bearing the moral
responsibility of what happened? If so, then these corporations inadvertently
have created an entity that is independent of the individual stakeholders and
their moral pangs. That is a very frightening prospect...

~~~
csours
It is not only responsibility but also knowledge. The ignition switch engineer
may not be aware of what exactly it turns on and off; even if that engineer
is, other engineers are not. (See page 1 of the GM Valukas report)

Analogy: does your DBA know about flushing to disk?

~~~
Too
Very good point. In huge systems such as a car everybody can not know
everything. But in this case, if the ignition switch team does not know how
their component affects everything about the car or how important it is the
responsibility should move up to whoever writes the specs on the ignition
switch. First of all for designing a system with a single point of failure in
the first place, secondly for not loudly notifying ignition switch team that
their component is vitally important and a single point of failure. As i also
said in a post above, way too much attention is given to the faulty switch
when this is actually a much larger system design error.

------
latch
Seems to be a lot of things at play, but what surprised me was the ability to
issue a modified part without a change to the model number. Seems like
something that could be enforced automatically? Like a version control system?
Are we running out of numbers?

~~~
bagels
Well, it's either a way to perform a coverup, or a cost savings by not needing
to make new tooling with the new part number on it.

~~~
mirkules
A new part number doesn't necessarily mean new tooling. However, you are
correct in saying that it is a way to cut cost by not issuing a new part
number. A new part number means:

\- Entirely new release/approval cycle. That means going through design,
release and QA stages (QA is extremely expensive in automotive industry).
Remember auto industry is not the software industry, there is no "agile" \-
waterfall is king because once the product is released, it is infinitely more
expensive to change it

\- Recalling old part numbers and shipping new ones

\- Even the cost of printing new labels and making new packing materials can
be non-insignificant

That said, a change in part equals a new part number, period. So, this was
very likely a coverup AND an improper cost-reduction exercise.

~~~
bagels
I agree with you.

I did happen to notice from pictures that these particular parts have the part
number cast on to them, so new tooling may be required.

~~~
mirkules
Ah yeah, good observation.

------
PhantomGremlin
Along with the other posters here, I am also "horrified".

But isn't this more-or-less the same at most/all other companies? They become
old and sclerotic and hidebound and resistant to change. Aren't most old
companies guilty of the equivalent of the "GM nod"?

    
    
       a staple of GM managers, nodding in agreement
       at steps that should be taken then doing nothing
    

Maybe companies like GM eventually run their course, and should be allowed to
die. Maybe we were wrong to bail them out in 2009? Would the country be better
off if a new company could take GM's place?

But, of course, GM couldn't be replaced "as is". It could only be replaced by
something more in tune with the times. It's only a {relatively speaking} tiny
company, but Elon Musk and Telsa could be "the next GM". Until Tesla
eventually runs its course!

~~~
djloche
Not one company should be bailed out. If a business fails, those who think
they can run that sort of company better should step in and buy the assets and
give it a go.

In the many or most cases where companies have built what has become core
infrastructure, regulations already exist for orderly shutdown and sale of
assets without disruption of service.

If a company becomes 'too big to fail', this means regulatory capture has
already occurred and you have far greater problems to resolve than mere
bankruptcy.

~~~
kefka
I disagree. Bail out big companies who fail miserably. THEN do the next step:

Take ownership of said company. If We're paying for it, We should control it.

------
yason
Still wondering why make physical ignition switches that actually mechanically
cut the flow of electricity.

For years, cars have been equipped with all kinds of 'convenience' electric
components such as Bluetooth, touch panels, and on top of that all critical
elements in the engine have been controlled by an ECU for decades -- yet still
only some cars have electrically, not mechanically, controlled ignition.

The car key doesn't even have to be wireless: you could still have the lock
and ignition key as now if you really wanted but turning the key would only
give a signal to the car's computer to wake up from sleep, authenticate the
key, close the main circuits electrically to let the components start drawing
the big current from the battery, and start booting sequence, self-tests etc.
and eventually drive current to the starter motor to crank the engine up.

That's a huge number of things to do, and it had best be left for the computer
to handle. Pulling the key from the physical ignition switch is aking to
starting and stopping your PC by pulling and inserting the plug from/to the
wall socket. You'd better push the suspend button and let your laptop put
itself to sleep and let it wake up by itself.

It's also much easier to design a system that will handle startup and shutdown
by itself, rather than a system that must survive opening and closing the main
circuits at any time.

~~~
cnvogel
I think there's a change happening, some new cars can be started by button-
press when the transponder (replacement for the key) is somewhere in the car
(e.g. in your trousers' pocket).

And then, of course there's the other aspect that in some cases you absolutely
want to be able to cut power to your engine. Wasn't that exactly the case with
Toyotas drive-by-wire throttles that apparently made people unable to stop
their cars?

Still: When you look at the electrical system for, e.g. a commercial airplane,
you'll see a dozen different independent power-supply busses that can be fed
from each other, different engines, or batteries. So that in case of power-
loss in one or a few of them, not all safety critical systems black-out.

I don't claim that such complexity is helpful or desirable in a car, but with
the majority of drivers relying on computerized systems it probably makes
sense to make power supplies more redundant.

{probably that's already happening, and I'm just unaware of it?}

------
a3n
These kinds of scenarios could appear in the "negative" column of a positives
and negatives analysis of whether to work in a safety critical, regulated
industry. Because even if an individual does his job diligently and ethically,
there is a risk that corporate motivations can coat you in shit.

Articles like this, and the Ranbaxy pharmaceutical article, make me wonder if
the satisfaction of working in safety critical industries is worth the risk.

------
kazinator
> _While Brooke’s lap belt glued her waist to the seat, her shoulder harness
> went slack the instant the engine shut off._

Jawdrop ... what? Those things should be purely mechanical, locked in place by
masses undergoing acceleration.

Shoulder belts requiring engine power is just monumentally stupid.

~~~
aidenn0
This was a case of 12V power disappearing to a lot of systems; that still
doesn't explain the slack harness, but electric pretensioners would certainly
fail to work in a crash under such circumstances.

------
mherdeg
If anyone else is feeling a sense of déjà vu in reading this article — there
was another in depth story about some of the technical details behind this
fault in the NYT in March 2014,
[http://www.nytimes.com/2014/03/29/business/a-florida-
enginee...](http://www.nytimes.com/2014/03/29/business/a-florida-engineer-
unlocked-the-mystery-of-gms-ignition-flaw.html) .

This Pando article has re-reported the story with lots of additional details
and does some great storytelling, so I'm glad to see it. Just wanted to add a
note for anyone who is wondering (like I was) "whoa, haven't I seen this
before?".

~~~
mreiland
I liked the storytelling as well, the only part that annoyed me was the way he
continued to drop back into the romanticizing of the characters. You've
already hooked me, I want to know what happens next, I don't give a shit about
the lifestory of the engineer, I only care that he's an engineer and how he
got involved in the story.

Other than that, I thought it was a great writeup.

------
csours
All sympathy to the families involved; what makes this the worst auto scandal
in history? Are there numbers for this or is this based on public opinion?

Disclosure: I have been a GM employee since 2013

~~~
viseztrance
I've read the article and this blew my mind:

"[...] if you took all of the cars that GM has recalled this year and lined
them up bumper to bumper, you’d end up with a line that would wrap around the
earth four times."

That's 30 million cars.

~~~
kefka
And if you took all the software that has been recalled, it would line up to
nothing... Because software doesn't have dimension.

And aside radiological software incidents (Therac 25 and similar), software
errors usually don't kill people. Engineering design failures usually do.

------
teh_klev
Previous HN discussion when this appeared in the NY Times:

[https://news.ycombinator.com/item?id=7492595](https://news.ycombinator.com/item?id=7492595)

------
sbhere
> 'Despite the conclusions of the police accident report, which claimed that
> Brooke had been driving too fast for the road conditions and lost control
> when she hydroplaned over a sheet of water...'

I would love to see an article on statistics of how many police reports are
factually inaccurate, even if only proven inaccurate by later
reporting/exposition.

------
gfwilliams
I'll probably get flamed, but...

I'm shocked that everyone here is so against GM on this.

Given the complexity of a car and all the things that could go wrong, this is
a very minor issue. The brakes still work (no ABS though), steering still
works (it's just harder to turn), and lights still work. It basically turns
into a car that's 20 years older, not a coffin on wheels.

The driver also knew there was a problem, wasn't happy with it, but still
drove her car regardless and then wasn't able to cope when it happened again.
She wasn't forced to drive it.

Maybe it's because I tend to drive older cars, but I'm all too aware that
they're machines and they break. I've had all kinds of things fail when
driving and I've had to try and cope with them safely. Luckily I have, but if
I hadn't been so lucky I wouldn't blame the car manufacturer.

100 years ago you could get killed by all kinds of things and it was more or
less just accepted - why are we now at the stage where a switch that's a bit
too easy to turn is the cause of all this blame?

~~~
stephenitis
Can't tell if this is supposed to be sarcasm.

I think the coverup or moral dilution at GM stands to be pretty damn awful.

engineering a failure situation and not playing out how it's most dangerous
scenarios might end is pretty bad. I don't think anyone would drive a car over
20 mph if they knew it would shut off and literally have to "cope" with the
stress and impending accident and hope to manually steer the car to safety.

I'm shocked you don't hold manufacturers to a higher standard.

~~~
gfwilliams
It's not sarcasm, I'm just very surprised that everyone's opinion is so one-
sided about this.

I think if you put yourself in the engineer's position, he probably didn't
believe that having a switch that was a bit easier to turn would lead to
anything more than mild inconvenience.

I think if GM honestly thought this would cause even one death they would have
changed the lock barrels. Even if you ignore the human side, the potential
costs damages from something like this would totally outweigh the cost of
changing them.

------
function_seven
The ignition switch in question seems totally unremarkable. I wonder why it
was redesigned in the first place? Why not use one of the many existing parts
GM had at the time?

Also—somewhat of a nitpick to an otherwise good article—the Bronco II is a
completely different model from the one driven by Al Cowlings during the O.J.
Simpson chase. That was the full-sized Bronco. When details like that are
wrong, it usually makes me suspect the accuracy of other facts in the story
(although in this case, the article seems otherwise accurate).

~~~
spydum
I also wondered this since the story broke.. Why do they redesign core
components like this, instead of just iterating on previous designs? I fight
this every day in the enterprise space. My only conclusion is ignorance:
sharing previous design or infrastructure knowledge is hard (or at least, not
prioritized). So folks end up recreating the proverbial wheel.

~~~
kbart
I guess that might be due to parts' supply and logistics problems. When
production counts in hundreds of thousands and millions, getting all parts
together and in time is quite a challenge.

------
S_A_P
Not really relevant to the article but bugs me because it was incorrect. The
Bronco II is not what OJ Simpson drove on his infamous low speed chase. He was
driving the full size Bronco. The bronco II was the mid size ford ranger based
suv(at least the article gets that right)

~~~
__david__
Perhaps you misread the article?

> _A cultural touchstone, the Bronco II was not only infamous for having been
> the little brother to the Bronco, which ferried a fleeing O.J. Simpson as a
> battalion of police cars and news helicopters gave chase on national
> television…_

That sounds correct to me.

------
markvdb
Waffle_ss , change your user name.

~~~
markvdb
I should have added some historical context to this request for those unaware
of it.

It's hard not to see the user name Waffle_SS as an unnecessarily hurtful
trivialisation of the Waffen-SS, the Nazi paramilitary force. See
[https://en.wikipedia.org/wiki/Waffen-
SS](https://en.wikipedia.org/wiki/Waffen-SS) for more information.

