
Toyota Unintended Acceleration and the Big Bowl of “Spaghetti” Code (2013) - CharlesFinley
http://www.safetyresearch.net/blog/articles/toyota-unintended-acceleration-and-big-bowl-“spaghetti”-code
======
Dav3xor
I've worked on safety critical software, for a couple different companies.
Code quality is a very real problem.

One company, the guy in charge of software was a CE, who would have been fine
as long as the hardware was at the level of sophistication as what he was
taught at school (He knew 8051 microcontrollers really well). He was really
good at giant switch-case statements. Function pointers were a little
newfangled and suspect.

Basically, he knew enough software engineering to get the hardware working.

That's one big problem that needs to be addressed -- there aren't a lot of
people being trained in the software side of embedded systems. You have CS
grads who for the most part aren't given much training on the low end of the
abstraction spectrum, and the opposite for CE people, so there tends to be a
very fuzzy area in the middle that causes arguments between the two camps.

The other company I worked for literally had crazy coding standards that
basically dictated 20,000 line functions and a bizarre sort of anti-DRY
mindset that I will never understand. You were encouraged to c-c c-v a block
of code, change one line, move on!

------
quant5
Patio11 has talked about how programmers are little respected and paid poorly
in Japan compared with the US. Could this have something to do with Toyota's
code quality being so poor and causing this issue?

~~~
Jemmeh
That seems so strange to me since I always think of Japan as being so
technologically advanced.

Granted, even here in the USA programming used to be considered "women's
work", mostly revolving around planning and organization. So they passed it
off to secretaries. I just didn't think that was something that was still
happening.

[http://www.smithsonianmag.com/smart-news/computer-
programmin...](http://www.smithsonianmag.com/smart-news/computer-programming-
used-to-be-womens-work-718061/?no-ist)

~~~
sosborn
> I always think of Japan as being so technologically advanced.

Ask anyone who has lived in Japan for a sufficient period of time and they
will laugh at this statement, especially as it relates to programming.

~~~
Jemmeh
I was thinking about robotics and consumer electronics mostly. I see that in
the news a lot. There's even a wiki page about it.
[https://en.wikipedia.org/wiki/Science_and_technology_in_Japa...](https://en.wikipedia.org/wiki/Science_and_technology_in_Japan)

~~~
sosborn
I can see how people think this, but once you live here you see the reality.
There are sectors though where Japan is still pretty damn advanced, but you
don't necessarily notice it in daily life. A great example is precision
manufacturing - they are awesome at it.

------
stretchwithme
These unintended acceleration incidents seem to happen mostly to elderly
drivers.

[http://www.theatlantic.com/business/archive/2010/03/how-
real...](http://www.theatlantic.com/business/archive/2010/03/how-real-are-the-
defects-in-toyotas-cars/37448/)

~~~
ebbv
And more importantly than that there are a couple of underlying facts that
invalidate the whole premise that unintended acceleration means automatic
accidents/death:

1) None of these cars are 700hp Supras that accelerate at drastic speeds.
These were normal consumer cars that frankly aren't that fast even with the
throttle fully depressed.

2) The brakes in all of these vehicles are many times more powerful than the
engine. Pressing the brakes would have stopped the vehicles, even if the
engine were attempting to accelerate full throttle.

In most cases I would never side with the corporation over average people. But
this is one of those rare cases where lawyers were able to hire dishonest
"experts" and snow over the judge and jury and get an unjust verdict.

~~~
UnoriginalGuy
Your two point list is silly. Consumer cars & SUVs can go 0-60 in under 10
seconds in almost all cases, and the fact that the brake can overcome that
acceleration is great, except it takes time, and within the time between
unexpected acceleration and stopping damage/injuries/death can occur.

I had a water bottle fall onto my right leg while turning, this caused a
sudden but short lived acceleration (e.g. 10 MpH of unexpected acceleration
for less than 1 second), this was enough to cause me to mount the curb, and do
some decent property/car damage (thank god nobody was on the curb). The brakes
worked perfectly, and stopped the car, but it does show how much damage even a
tiny burst of unexpected acceleration can do (the entire incident was under
5-10 seconds).

It honestly boggles my mind that someone can dismiss any length of unexpected
acceleration because "cars [aren't] 700 hp Supras." Consumer cars are plenty
powerful enough to cause injury and death due to any length (even 1 second) of
unexpected acceleration at the wrong moment. Sure, if you're going straight on
the freeway then you aren't going to even notice, but in a car park, while
turning, or stopped at a school crossing, the brake's ability to overcome the
engine are largely irrelevant since you won't be expecting it.

~~~
kllrnohj
> Consumer cars & SUVs can go 0-60 in under 10 seconds in almost all cases,
> and the fact that the brake can overcome that acceleration is great, except
> it takes time

The actual data paints a different picture:
[http://media.caranddriver.com/images/media/51/braking-
result...](http://media.caranddriver.com/images/media/51/braking-results-
photo-319716-s-original.jpg)

That said people don't do well with surprises so while the car mechanically
would have no trouble at all stopping very nearly as fast due to a wide open
throttle, the driver is probably freaking out instead of just slamming on the
brakes.

Also people suck at actually slamming on the brakes, regardless of the
situation. It's not something people want to ever do. They'll hit the brakes,
they just won't go anywhere close to really pushing that pedal to the floor.

~~~
UnoriginalGuy
> The actual data paints a different picture

To what? That doesn't contradict what I said at all.

~~~
drbawb
All the cars added a marginal [ O(10ft) ] amount of braking distance at wide
open throttle; with the exception of a very powerful, heavily tuned sports car
which was the only one able to overwhelm its brakes.

If we convert MPH to FPS we can look at the data as follows:

70 MPH: 102.667 fps 100 MPH: 146.667 fps

    
    
        CAMRY, V6,  70-0: delta 16 feet,  .15sec
        CAMRY, V6, 100-0: delta 88 feet,  .60sec
    
        INFINITY  70-0: delta  9 feet, .08sec  
        INFINITY 100-0: delta  6 feet, .04sec
    

We're talking tenths or hundredths of a second of unintended acceleration. A
cursory search tells me reaction time to visual stimuli is about ~250ms.[1][2]

This means that in all but the Camry 100-0 case your reaction time is a larger
factor in braking distance than the presence of wide open throttle.

[1]:
[http://www.humanbenchmark.com/tests/reactiontime/statistics](http://www.humanbenchmark.com/tests/reactiontime/statistics)

[2]:
[http://www.jneurosci.org/content/26/15/3981.full.pdf](http://www.jneurosci.org/content/26/15/3981.full.pdf)

------
achr2
Isn't it time for an oversight agency that looks at code quality within
safety-critical systems? The FDA forces oversight of drugs that less than
100,000 people a year will take. Yet there is no agency looking into the
software quality of a car management system that millions of people will drive
with every day.

~~~
bradfa
This will add a tremendous amount of cost to product development. This might
be worthwhile if simple consumer or industry pressure doesn't work.

~~~
arbitrage
> This will add a tremendous amount of cost to product development.

Not that many people are getting killed, so it's an acceptable tradeoff?

~~~
hga
Yes, it's an potentially acceptable tradeoff. While the figures vary,
economists and the like have estimates of how many people you'll kill for
every N million dollars you extract from the economy (and they're below 10
million last time I checked). Just how much do you propose to extract with
such a regime?

------
oldpond
I am quite surprised by this, especially since it comes from Toyota, the
company that invented Lean. This is very relevant given that the debate over
upgrading vehicle software is raging on and the Volkswagen scandal has not
settled just yet. Too bad we can't crowd source the various automobile
software development teams. I am curious to see if any of them are any good.
In my travels over the last 10 years I have encountered very few skilled
software development teams at the enterprise level. Given the current state of
the industry I would not e surprised to find them all wanting.

Thanks for posting.

------
sakopov
One of the general principles of engineering ethics states "A practitioner
shall, regard the practitioner's duty to public welfare as paramount."

I think part of the problem is that software development isn't considered an
engineering discipline and code of engineering ethics goes out the window.

~~~
burnallofit
I got a degree in CS at UC Berkeley. There were exactly 2 lectures on ethics,
both given by the same professor. Kudos to him (Ousterhout), but shame on the
department. I'm not singling out UCB, I'm sure many other schools have the
same problem.

------
swiley
Critical systems should be open source by law.

~~~
hoorayimhelping
I love magic bullets! As we've recently seen with things like Heartbleed, open
source has no vulnerabilities, no bugs, no errors, and is free from all the
problems of proprietary software.

~~~
Jtsummers
One of the problems with embedded systems like this is that very few eyes ever
see the code. Even in more regulated fields like avionics. The code will be
examined primarily by the developers, testers, their QA and maybe the company
that hired them if it's contracted out.

Instead, what gets examined are the artifacts like requirements documents, the
results of tests, specifications and such. This allows poor code quality to be
hidden, and, to some extent, encourages sloppy development practices
(especially when time is critical). Exposing the code to more people will have
several effects, but the main two (from my perspective) are:

* Developers won't release as much bad code, either due to pride or insistence from their management.

* Bugs _may_ be more easily discovered and diagnosed if the code is available. As it is now, it's a black box. So if I find an issue I may be able to repeat it, but I can't examine the code to see why it's actually happening or to correct it.

------
TerraHertz
I'm a retired electronics design engineer and embedded programmer, and I will
NEVER own a car with any kind of vehicle/engine management computer. Old cars
for me, forever. I flatly refuse anything but fully manual and direct
mechanical gears, clutch, steering, brakes and throttle.

Curiously the chief engineer I knew at a major car service center, also felt
the same way.

And that's not even touching on the insanity of building computerized vehicle
systems with always-on GSM data links to the Net. Ask Michael Hastings how
that worked out for him.

Also I agree that critical systems software should be legally required to be
open source.

~~~
benihana
I love the absurdity of this and arbitrariness. You'll happily drive in a car,
one of the most dangerous machines people use regularly. But if it has a
computer in it, no siree, that's when things get too deadly to deal with. All
of the other thousands of moving parts, like the thing that takes energy-dense
hydrocarbons and ignites them several thousand times a second in hot, high
pressure tubes - that's fine and totally safe. It's the ECU that makes the car
dangerous. The fact that the only thing separating you walking on the sidewalk
from death from a two ton metal box is the convention that we'll all stay
within the lines painted on the ground. That's fine. It's the ECU that you're
afraid of. Absurd.

~~~
Spooky23
It's not absurd at all. For the questionable benefit of the ECU, you get a
black box system that may or may not be garbage controlling the primary engine
input, that may or may not fail safe. Give me the thing that grandpa designed
75 years ago.

In the olden times, the throttle was controlled by a mechanical device and
tensioned springs. The failure characteristics were studied for 150+ years,
and the state of the mechanical components could be assessed by visual or
physical inspection. The failure scenarios for open throttle are also non-
obvious things to workaround. What do you do? Pump the brake? Take the car out
of gear? Depress the accelerator to reset? Turn the key? It's a complex
decision matrix with life-and-death consequences, and the correct answer will
vary by car configuration and vendor.

The ridiculous positions taken by posters here are indicative of how
engineering fail like this happens.

~~~
drbawb
You speak of the olden times like they're long gone? My car is from 2001, has
side curtain airbags which will render most common crashes non-fatal, and it
still has a fully mechanical throttle and no electronic brake controller of
any kind. I don't consider it all that old.

Yes it has an ECU, but EFI is not the problem in my opinion, and the computer
by itself doesn't frighten me. EFI was a fantastic invention as far as I'm
concerned. Also despite it being a "black box" I find it much more pleasurable
to tune and maintain EFI systems over fickle carburetors.

The real problem was making the ECU an _active control system_ which directly
controls the engine, throttle, brakes, etc. in response to your inputs; as
opposed to a passive one which merely _reacts in response to changes in its
environment_ (e.g: more air moving through the intake, wheels locked up,
losing traction on one side.)

So yes, my '01 Toyota has a black box, but it's simple enough that it could be
replaced by a handful of aftermarket controllers, many of which have their
source freely available, or available for a modest licensing fee.

\---

Also I'd like to disagree that reacting to WOT is a "complex decision matrix."
\-- My instinctual reaction would be as follows.

First you open the clutch and/or put the car in neutral. Disconnecting the
motor from the wheels is the most reasonable solution to this problem. When I
was taught to drive stick the very first thing I was told, before I ever moved
the car an inch, was: "when you need to stop, clutch and brake."

(Of course if it's an automatic transmission: "going into neutral" is just
controlled by another black box. Sucks to be you if you hit deadly bugs in two
separate powertrain management controllers.)

(As an aside I do personally know people that commute every day in the US, and
they don't even know what a transmission _does._ Why are we licensing these
people as skilled motorists?)

If I somehow found myself without even the most basic control of my
transmission then you just press the brakes as hard as you can and you stop in
~300 feet.[1]

If that didn't work, or if I had stopped but hadn't regained control of the
vehicle, I would then kill the ignition. (To be fair: I'm told this is not
quite so simple in modern cars! Apparently someone thought "pushing and
holding a button for 3 seconds" was a better idea than "turn a key." \--
However I also wouldn't agree to drive a car if I didn't know something as
basic as how to kill the ignition under duress. I'm the sort of guy that reads
the manual cover to cover for fun.)

If killing the ignition doesn't work[2] and your transmission is somehow stuck
engaged then today is really not your day.

I don't see how any of this requires any more skill than driving does
normally. To me this is not some complex decision tree, it's reflex at this
point.

(Also there is a good reason I would brake before killing the ignition. Brakes
and steering are mechanically assisted by the engine. It would be extremely
irresponsible to cut the ignition in a vehicle w/ power steering and power
brakes on a public motorway in my opinion. -- Again I don't think this is some
complex decision, I believe it should be requisite knowledge for being
licensed to operate a motor vehicle under such conditions.)

tl;dr: the complexity in this matrix is inherent in the task itself. If this
is "too complex" then maybe we should work to improve our driver training and
licensing programs; or better yet consider having more people take public
transit, instead of handing out licenses like candy.

[1]: [http://media.caranddriver.com/images/media/51/braking-
result...](http://media.caranddriver.com/images/media/51/braking-results-
photo-319716-s-original.jpg) [2]:
[https://www.youtube.com/watch?v=3NRaqgab0_w](https://www.youtube.com/watch?v=3NRaqgab0_w)

~~~
hga
_Apparently someone thought "pushing and holding a button for 3 seconds" was a
better idea than "turn a key."_

You know, that's not _completely_ insane:
[https://en.wikipedia.org/wiki/General_Motors_ignition_switch...](https://en.wikipedia.org/wiki/General_Motors_ignition_switch_scandal)

Although, wow, that's an _awful_ article, skimming it there's only this hint
of the root cause: "After being asked by Missouri Senator Claire McCaskill
whether a GM engineer had apparently lied under oath, [GM CEO] Barra confirmed
that this had indeed happened (or at least seemed to)." The problem, besides
GM having a procurement system that assumed people in it wouldn't lie through
their teeth about lethal problems, was a single engineer who selected an out
of spec switch, and then, for example, slipstreamed a better one into the
system without a part number change.

(Otherwise we're in total agreement.)

------
Jansemon
How can breaks not stop the engine?! Parking break being a different thing,
but the main ones?

~~~
delinka
"Brakes" are at the wheels. The only time stopping a car from rolling also
affects the engine is when everything remains engaged (manual transmission,
manual clutch; in gear, clutch not depressed to the floor.) In a vehicle with
an automatic transmission, the transmission begins to disengage from the
wheels as the wheels spin slower than the engine is pushing. (Very simplified
explanation...)

i.e. brakes on cars are not designed the stop the _engine_ but to absorb the
vehicle's momentum.

~~~
radiorental
Also, it's been reported that drivers pumped the brakes in a panic (and
understandably so).

This will have the effect of reducing the power assist, which is already
reduced due to high revs / low vacuum.

~~~
jmhobbs
I wonder if this is a behavior inherent to older drivers? I'm in my 20's, but
my first car was from the late 60's. I hadn't properly bled the master
cylinder on my brakes, so I would have to pump them to make them more
effective. Just wondering if it's a natural panic behavior, or something
learned.

~~~
drbawb
It's something learned. You are explicitly taught to do it if your vehicle
does not have ABS. It prevents the wheels from locking up. ABS does the same
thing, and does it much more effectively than a human ever could.

Preventing the wheels from locking up under hard braking is crucial to
stopping when you have little traction. To provide traction on any surface
_wheels must keep rolling._

When wheels are static their contact patch is effectively the size of a hockey
puck, that little bit of rubber is not very good at stopping a car going
80MPH. Not compared to disc brakes w/ ceramic pads bleeding off all that
energy, at any rate.

While I'm on the subject, I'll take this time to drop a PSA: on ice, where ABS
is most helpful, the rubber of your all season tires is about the consistency
of a hockey puck. -- Please invest in actual winter tires if you get regular
snowfall.

(Also if you live in Texas: invest in a set of winter tires anyways and go
have a blast when the streets are deserted.)

------
illumen
It's interesting that VW cars have been taken off the road for unintended
acceleration. Also that quickcheck was used to find many timing related bugs
between software components that VW uses. It was used after the models were
taken off the road. There are other models still not recalled using similar
software from the same era.

~~~
SixSigma
Almost all brands of cars have. You should read the NIST database sometime -
the list of recalls is much longer then the ones you see in the news.

While you're at it, read the FDA list of food and medical recalls too, so much
risk!

------
Splines
A big takeaway that I got out of this is that nearly everybody forgot about
this.

People are still buying and driving Toyotas.

When the VW emissions thing started making its way through the news cycle, I
read comments postulating that this might be the end of VW. Hah - people are
going to forget about VW just like they forget about everything else.

~~~
csours
> When the VW emissions thing started making its way through the news cycle, I
> read comments postulating that this might be the end of VW. Hah - people are
> going to forget about VW just like they forget about everything else.

It wasn't the end of Toyota, and it won't be the end of VW; but it was the end
of Toyota's goal to be the Biggest Car Company in the World[1][2], and it may
be the end of VW's goal to be the Biggest Car Company in the the World.

Being #1 volume vehicle manufacturer is a curse!

1\. [https://hbr.org/2007/07/lessons-from-toyotas-long-
drive](https://hbr.org/2007/07/lessons-from-toyotas-long-drive) \-
Interestingly, in this article Mr Watanabe claims that Toyota never had the
goal to be the biggest car company.

2\.
[http://www.economist.com/node/15576506](http://www.economist.com/node/15576506)
\- James Womack, one of the authors of “The Machine that Changed the World”, a
book about Toyota's innovations in manufacturing, dates the origin of its
present woes to 2002, when it set itself the goal of raising its global market
share from 11% to 15%. Mr Womack says that the 15% target was “totally
irrelevant to any customer” and was “just driven by ego”. According to Mr
Womack, the requirement to expand its supply chain rapidly “meant working with
a lot of unfamiliar suppliers who didn't have a deep understanding of Toyota
culture.”

Disclosure - I work for GM, these opinions are my own, etc.

------
lurkinggrue
Just dealing wit the entertainment messy of modern cars I already came to the
conclusion that the car industry has no damn clue how to write software.

It's like the worst internal shit code you see in the in house tools at many
companies where the moment it ran it was considered done enough.

------
mfoy_
>Toyota had more than 10,000 global variables.

Oh my.

