

Blame for 'switch from hell' falls heavily on one GM engineer - r0h1n
http://europe.autonews.com/article/20140605/COPY/306059903

======
SilkRoadie
It doesn't matter what the engineer did. The fact that the car company had so
few controls that a single engineer could approve something faulty then change
the specs of it in future models without anyone noticing is worrying to say
the least.

It is a bit like banking and these "rogue traders" who spend a billion
dollars. They are at fault certainly BUT more so it is the banks fault for
having such lax measures that they had access to this much money.

~~~
spindritf
They had checks in place for errors, not for being deliberately misled by one
of their own employees. If they had checks against that, we'd be reading a
blog about procedures and corporate culture in GM suffocating engineering.

~~~
bilbo0s
Being a pretty big process guy... I have to agree with SilkRoadie here. I
should preface this by saying that my experience comes from developing FDA
regulated medical diagnostic software and not automobiles. So I realize there
may be a significantly lower quality threshold that GM is obliged to meet.
Having mentioned that, I can tell you that, at least in that medical
diagnostic software startup, the idea that an engineer could design and
implement ANYTHING and get it into the software build without QA noticing is
somewhat fanciful. Someone from QA would have had to be in on it.

The process and company structure pretty much made it impossible for the the
type of situation outlined in the article to occur.

Just as a 30,000 ft overview of the process... imagine a place where every
little change required a minimum of 4 signatures, engineer, QA engineer, Head
of QA/Reg, and CEO. This allows not only QA, but also the FDA to determine
precisely who was responsible for each change. Imagine further that the
signatures have to be on paper, and that giant mass of paper documentation has
to be submitted to the FDA, along with mountains of other documentation as
well as the software itself, for approval. (But everyone understands that it's
also submitted so that the FDA knows who to throw in prison if serious bugs
crop up.)

The situation outlined in the article should be STRUCTURALLY impossible as
well. At the startup I worked at for instance, build engineering was under QA.
This meant that a development engineer could not give the build team a build
directive. Only the head of QA/Reg could do that. Also, the head of QA/Reg
didn't report to the COO or CFO or Chief Counsel or anything like that, she
reported directly to the CEO. As well, she was a frequent contributor at board
meetings. There was just no way to pull an end around on her.

That's just a rudimentary outline of the controls we had in place at that
startup. Believe me, it actually got a WHOLE lot more restrictive than what
I've outlined, but the other stuff is not important to the point I'm trying to
make.

And that point is this...

While I realize that for cars the safety standard is probably lower than it is
for medical imaging and RTP packages, you would think GM would have those
rudimentary controls in place at a minimum ???

~~~
benjamincburns
Unless you've got a series of physical safety measures (think GoldenEye style
dual keys), your processes are just a series of rules. Enforcement of these
rules is probably quite redundant, but I promise you that with sufficient
motivation someone high enough in your company could knowingly insert (or
cover up) flaws which could kill somebody.

Such motivation might come from the fear of losing your job.

~~~
bilbo0s
Well... yes... we had "dual key" type things going on as well. (That was part
of what I meant by "it got a WHOLE lot more restrictive".)

But, really, isn't an "engineer", someone pretty LOW in the hierarchy ?

I mean, that's kind of the point, the better the structure and processes...
the higher up you will need to go to subvert them. In the case of GM... the
processes were so poor that you didn't need to go beyond the engineer level to
subvert them. (Assuming the information in the article is to be believed.)

~~~
cmdrfred
I imagine that it may have been somewhat simple for GM to implement a system
to prevent this, but from another perspective I also can conceive that such a
system might effect GM's bottom line negatively. If having engineers cut
corners on their own (likely with the encouragement of management) saves
money, incompetent upper managers might pad their bonuses with these little
cut corners(an out of spec switch here, a faulty connection there). Normally
this just means small problems for consumers down the line, this time it just
ended up killing people. So who gets blamed? The guy at the bottom? I'm not
buying it.

------
pjmorris
"A loss of X dollars is always the responsibility of an executive whose
financial responsibility exceeds X dollars."

\- Gerald Weinberg's 'First Principle of Financial Management' and 'Second
Rule of Failure Prevention' [1]

[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald
Weinberg, Dorset House Publishing, 1993

------
seren
While interesting, I find the narrative pinpointing the issue on a single
engineer behavior a bit too convenient to be believable.

~~~
fleitz
Why? Have you never fixed a bug in your own code, or not followed every single
one of your companies' procedures?

~~~
seren
I am a SW engineer currently working in a regulated industry, and there is no
procedure in place for me to fix something quickly behind everybody's back.
The code should be at least reviewed, integrated and tested by 3 independent
people. And the issue and the resolution action would have been
reviewed/signed off at the system level as well. So it would be very unlikely
that I could hide something for years without suspicion.

This is in a way cumbersome and annoying but it is understandable why it is
put in place this way.

I used to work in consumer electronics, where I could commit whatever I wanted
without any kind of supervision or checks, but this is generally not something
you can do when it is a matter of life and death.

~~~
TheSpiceIsLife
That being the case, what is your opinion of what is going on here?

~~~
seren
It is possible a single engineer was negligent or even malicious, but I would
rather believe the whole organization was at the very least encouraging a
culture of "do whatever to be ready at the end of the quarter and I don't want
any detail" (orally of course)

You can't be reckless in a place where everyone is paying really close
attention to safety and regulations. On the other hand if you are in
environment where everyone is pretty lenient, it is much easier to slip
between the checks.

But, from a journalistic point of view, even if the story is not fabricated,
it is a better story to have a villain, than trying to convey to the
readership how complex organizations (mis)behave.

------
funinobu
This news story talks about the proximate cause of the failure and how the
cover-up started. But it doesn't say how the cover-up propagated. 14 other
people were fired. But we don't know why or whether that was the extent of
either covering up or incompetently failing to to uncover either the problem
or the cover-up.

------
leoedin
Does anyone with experience in the automotive industry have any thoughts on
this? What sort of culture would lead to the selection of an inappropriate
part in the first place? Would it be cost driven? Does this idea that an
individual engineer's component selection decision would play such a big part
ring true with other commentators experiences?

~~~
valarauca1
I've actually lived with a DRE for several months, and he's a drinking buddy
of mine (works at GM too). The first thing you need to understand is the
pressure DRE's are under.

It sounds odd, but General Motors is a large and complex entity. Their release
system is old and buggy, not to mention never completely overhauled, just
constantly having more checks ticked on when ever a recall happens. This is
actually true at Ford too, the Big 3 hold on to technology a lot longer then
other companies. Ford's time clock system is still ran on emulated System360's
using MTS a circa 1967 OS.

Lastly DRE's are more of a projects internal sales arm (they _sell the
product_ to management. A DRE's job is to also get the test results from the
test engineer, get it to the management, take the managements complaints to
the designers/hardware engineers, and relay the tests plans back to the test
engineers.

Basically high technical middle management. Their responsibilities at some
companies may involve actually writing test plans, sometimes not. I don't
believe GM does directly but somebody under the DRE does this.

Ultimately the DRE is just the guy who gets all the forms to all the right
people so he can push the plans into the system. And takes all the
responsibility when a product fails. On top of that your normally managing 5-6
products concurrently, each taking 18-72 months to clear the system. Your work
load is basically endless meetings 500+ email a day, and 100+ phone calls.

~~~
swimfar
DRE=Design Responsible Engineer

~~~
valarauca1
Design/Release Engineer. Its just a middle management fall man who knows CAD.

------
dankohn1
I think modern testing culture and source repositories can help avoid this
kind of negligence in software projects today. I'm hardly Panglossian enough
to suggest that it's common, but if you have an engineering culture where
specifications and especially regressions are encoded in tests, it starts
becoming clear when your software project is or is not satisfying the
requirements. Decisions like removing a test because it fails intermittently
are documented by source control, and can be reviewed by peers and management.

------
xyzzy123
I have to wonder how much the focus on the switch is papering over other
problems. The fact that a non-essential mechanical component can cause
accidents like this seems fragile.

When the key pops out, the engine stalls, power steering goes out and the
airbags do not inflate. Where is the "defense in depth"? A robust system
should really require _two_ things to go wrong.

Is it that "key in ignition" is a critical part of the vehicle's state
machine?

~~~
hga
Pretty much, I think.

 _Not_ being able to decisively turn off the engine has also killed people,
remember the Toyota issue not long ago? We think that in general the drivers
panicked and floored the accelerator thinking they were pressing the brakes,
but the complexity/difference of turning off the engine with a button---as I
recall it had to be held down for something more than a second---vs. turning a
key all the way counterclockwise and ultimately pulling it out could have
resolved the user errors short of fatal crashes.

Ditto the theory that some or many of these were caused by the accelerator
getting stuck on a floor mat. Or an embedded computer and/or components
attached to it failing in a bad way.

Compare to the big red buttons on some computers and hopefully in all big
machine rooms: there are times, when lives are threatened, that you want an
easy way to immediately turn everything off. Even if you add a molly-guard in
front: [https://en.wikipedia.org/wiki/Big_red_button#Molly-
guard](https://en.wikipedia.org/wiki/Big_red_button#Molly-guard)

~~~
deserted
If you're driving down the freeway, putting the gear selection lever to
neutral would be a much better choice than killing the engine and taking out
the key.

~~~
hga
If you're so panicked you're flooring the accelerator thinking it's the brake,
you might not think of that, and since you've not actually depressed the brake
pedal, it wouldn't shift anyway, would it?

If you're truly standing on the brake, then it ought to work, but you might
not think of it. I'll bet a lot of people don't usually think of any gears but
park, normal forward and reverse.

~~~
deserted
On my car, you can always shift from Drive to Neutral just by pushing forward
on the gear lever, without your foot on the brake, without pressing the button
on the gear lever, etc.

------
dmritard96
Its interesting how much attention the general public has given to this yet
for only a handful of deaths. For some perspective, 34,000 people died in the
US during 2012 due to motor vehicles[1].

The big automakers are ultimately causing way more deaths by not adopting
autonomous vehicle technology sooner if you ask me. And they will continue to
do so because they realize that autonomous fleets make more sense than every
consumer buying a car or two hence autonomous cars mean their business is in
jeopardy/going to shrink.

We might also look into deaths associated with air quality/pollution and the
number of deaths resulting from it. Its always going to be hard to pin down
what number of deaths correlates to how many cars driving some number of
miles, but I would put serious money on lower emission vehicles significantly
reducing deaths resulting from air pollution. Detroit's reluctance to adopt
cleaner technology and its obvious love affair with oil is killing many more
than their incompetent switch DREs.

Ultimately, even the switch design is a similar phenomena. While many
automakers started to get away from keyed ignition switches altogether, GM has
continued to use a super cheap part that's obviously a design from the past.
You can't tell me the radio on my grandmothers key ring cost versus their
switch is worth 13 lives.

Certainly, don't mean to trivialize those that have been killed by this, just
want to point out that Detroit is so slow to change that ultimately its
costing us way more lives than just those killed by this switch problem.

1
[http://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in...](http://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in_U.S._by_year)

~~~
chris_wot
I disagree. This is particularly interesting because there was an active
coverup of the issue. This could have been resolved with almost NO deaths, had
the actions of this one engineer not been taken.

~~~
dmritard96
I don't mean to troll. I agree its interesting because there is a cover up and
an engineer who could have saved 13 lives and didn't.

My main point is that big automakers are doing this same think, cover up and
all on a grand scale but because you have to look at it from 10,000 feet to
see it, it doesn't get the same type of criticism.

you said you disagree, there were like six points I tried to make, was there a
particular set you disagreed with?

~~~
chris_wot
I never considered your post a troll, I merely disagree with the general
premise that I took from your post that this issue is less important than
tardy development of autonomous vehicle and emissions controls technology.

Those are important, but not quite as immediate as this matter, is really the
issue I had. But you are quite right, those are still important issues.

(shame on those who voted you down, IMO you made a good point but dialogue
would be more appreciated than random clicking on a down vote).

------
puppetmaster3
Blame the engineer. Hmm. Lets see. Will I ever buy Government Motors in my
life? That would be a no.

~~~
jusben1369
Kind of disappointing to see "Government Motors" on Hacker News.

~~~
shiven
Giant Monoliths are (understandably) not looked upon kindly on HN. Move fast &
Break things is the mantra here. ;-)

~~~
jusben1369
I hear you. But "Government Motors" is a clever marketing phrase right along
with "Death Panels" that became a catch cry for a political movement high on
rhetoric and low on intellectual rigor. Thus my disappointment that someone on
HN might also inhabit that camp.

~~~
shiven
_Thus my disappointment that someone on HN might also inhabit that camp._

Yeah, I hear your disappointment. But, HN is not (and doubt ever was) a
homogenous community. It would be pretty boring and even more of an echo
chamber if it were. Roses and Dandelions, one more than the other ;-), all
form the HN crowd! Sure, dissing reddit _ization_ is a part of being a
responsible HN citizen, but let's not throw the baby out with the bath water!

(now get off my nice lawn all you pesky whippersnappers) :P

