
Fatalities vs. False Positives: The Lessons from the Tesla and Uber Crashes - szczys
https://hackaday.com/2018/06/18/fatalities-vs-false-positives-the-lessons-from-the-tesla-and-uber-crashes/
======
evrydayhustling
Great points regarding aligning precision/recall of AI systems with actual
human supervision capabilities. One quibble: I hate seeing articles
uncritically repeat Uber's line that a human driver could not have reacted in
the same scenario. The dashboard cam footage they released does not accurately
represent either human or car perceptual capabilities [1], and folks
recreating the scene have shown that it doesn't even look like a good (or
unaltered) example of what a dashcam should have perceived [2].

[1] [https://ideas.4brad.com/it-certainly-looks-bad-
uber](https://ideas.4brad.com/it-certainly-looks-bad-uber) [2]
[https://dgit.com/residents-recreate-crash-uber-car-
case-5616...](https://dgit.com/residents-recreate-crash-uber-car-case-56161/)

~~~
mark-r
There's no way of knowing if a human driver could have reacted properly in
this specific scenario. However if you live in an area where deer frequently
cross the road, you'll know that collisions with deer occur regularly; that
would indicate that it's easy to hit things that cross your path, even when
you _should_ be able to see them.

~~~
jimbofisher1
The human operator was responsible for braking in this situation and failed to
do so.

[https://www.ntsb.gov/investigations/AccidentReports/Pages/HW...](https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18MH010-prelim.aspx)

~~~
mark-r
That was my point, human operators are responsible for not hitting deer yet it
happens every day. The only reason it doesn't happen to people more often is
that people are better at avoiding running in front of cars.

~~~
romwell
>That was my point, human operators are responsible for not hitting deer

How is this even comparable?

A deer jumping in front of a car is not the same as a person walking the
bicycle slowly across several lanes (and almost making it to the other side)
on a rather well-lit road.

~~~
zaarn
Because the human is currently monitoring the car to not hit a deer, as other
comments mentions, it can take dozens of seconds to regain full awareness.

~~~
jimbofisher1
Dozens of seconds? Have you ever even driven a car? lol

~~~
zaarn
Yes, dozens of seconds. This is the scenario of "human was actually doing
something entirely different because the car is self driving"

This is _NOT_ classical reaction time.

------
ska
This set of trade-offs in self driving cars is exactly the same one that CAD
(computer aided detection/diagnosis) systems have been making in medicine for
decades. In many cases both type I and type II errors will (with statistical
certainty) kill people over enough iterations.

It's not an easy problem, but the best you can do is demonstrate that the
system improves on the standard of care. In other words, overall the trade-off
has better outcomes than not using the system. The same will be true for self
driving cars if/when they reach mainstream use. The important thing is the
average performance is improved significantly.

It's worth noting that, like the CAD systems, focusing too much on fixing an
individual error can cause a degradation to the overall system. As these
things get better and better, you'll have to be much more careful about
applying non-obvious fixes, or risk a much worse outcome that doing nothing at
all.

~~~
rayiner
Imagine a CAD system that was so fundamentally unreliable that it ignored
positive hits because the high rate of false positives otherwise would make
the system useless. Instead, the CAD system came with a warning printed on the
instruction manual that a doctor had to review every scan anyway.

~~~
bsder
Welcome to breast cancer screening.

The current systems are so good at detecting tiny anomalies that probably
don't matter that their usage has had to be tuned down.

~~~
ska
You have to be very careful with false positive rates with something like
breast screening, as the work up rate has definite negative consequences.
Radiologists have too high a false negative rate on such screens, so you're
trying to achieve a balance. To be really blunt about it, you are trading off
complications/deaths from missed cancers against complications/deaths from
biopsies, but these categories occur at very different rates.

------
niftich
We're asking a lot from this kind of software (for good reasons), but humans
commit hundreds of leaps of faith of various severity on the roads daily --
failure to yield, failure to maintain a safe following distance, assuming
other drivers immediately adjacent to you, like in lanes that are
significantly slower than yours, will keep driving safely and carefully.
Urban, peak-hour traffic on most US freeways is an exercise in collective
insanity, riding people's bumpers at 55+ mph (and often significantly higher),
leaving little room to stop for incidents [1][2][3][4] or debris.

But only a small subset of these situations result in significant accidents,
because unimpaired humans, largely, have some intuition for self-preservation.
On the other hand, we're expecting an algorithm coded by humans to perform
better than a complicated bioelectric system we barely understand.

Waymo's self-driving program has opted to thoroughly understanding its
environment, which is why their cars drive in a manner that bears no
resemblance to how humans actually drive. We as a society have to eventually
reconcile the implications of the disconnect.

[1] Unsafe lane change in traffic with different lane speeds:
[https://gfycat.com/CleanGleefulArawana](https://gfycat.com/CleanGleefulArawana)

[2] Tailgating causes crash, swerve, multi-vehicle accident:
[https://www.youtube.com/watch?v=j0rj2sZ1KA4](https://www.youtube.com/watch?v=j0rj2sZ1KA4)

[3] Inattention to incident causes further accidents:
[https://www.youtube.com/watch?v=hZL6OKwQGew](https://www.youtube.com/watch?v=hZL6OKwQGew)

[4] Inattention in slowing traffic causes accident:
[https://www.youtube.com/watch?v=Ff7wbSwTuEk](https://www.youtube.com/watch?v=Ff7wbSwTuEk)

~~~
jakelarkin
human drivers routinely do things that are not justifiable by vehicle physics
(e.g. stopping distances) because they have a comparatively sophisticated and
unconscious understanding of our physical world, how changes may develop and
how other human drivers or pedestrians might react. self-driving is a much
harder and unlikely problem than the perpetual hype-cycle is willing to admit.

~~~
std_throwaway
Mostly because they use the heuristic of "see, nothing happened last time" and
do it again but with a little less safety margin.

~~~
nradov
It's partly that, but also a matter of human drivers trusting each other to
some extent. In congested areas if you leave the recommended safe stopping
distance then another driver will always cut in front of you. So people adapt
by closing up the gaps and hoping the drive in front doesn't do anything
stupid.

~~~
viraptor
I find this one interesting. I rarely drive in the city, so when I do the
distance in queues seems pretty crazy. But at some point I stopped caring that
people cut in in front. The active cruise control keeps my distance now (apart
from when we're stopped) and often people will use that gap. But... so what?
Let them and keep your distance again. It really improves the experience.

~~~
std_throwaway
If you try to account for it, it's not much time at all. And you're improving
traffic flow by providing a means to change lanes (a bit more) safely. Most
people "cutting in front" are actually just in search for a spot.

------
Semirhage
_If a company is playing fast and loose with the false negatives rate, drivers
and pedestrians will die needlessly, but if they are too strict the car will
be undriveable and erratic. Both Tesla and Uber, when faced with this
difficult tradeoff, punted: they require a person to watch out for the false
negatives, taking the burden off of the machine._

Companies should not be allowed to punt this way, when lives are on the line.
This really is a problem in need of a regulatory or legislative solution, as
multiple companies prove they don’t have anyone’s interests at heart, but
their own. Worse, every pedestrian, cyclist, and other driver on the road who
didn’t sign up for this in-the-wild alpha is being drafted to enrich the likes
of Tesla and Uber.

~~~
ckinnan
Lives on the line? On average, existing cars kill more than 100 people each
day on American roads. The current system of human-driving cars has killed one
million Americans over the past couple of generations, mostly from human
error. To save lives we need to push forward on self-driving tech.

~~~
pythonaut_16
No, on average existing _drivers_ kill more than 100 people each day
[source?].

There's no proof yet that self driving cars are safer than human drivers. And
more importantly there's a big difference between being safer on average than
a human driver, and being safer than the average human driver.

~~~
francisofascii
The average was 102 per day in 2016, and that is just in the U.S.
[https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...](https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in_U.S._by_year)

------
simion314
I can still see the people commenting that the Tesla driver did not had his
hands on the wheel and he was warned when this warring happened 15 minutes
before the accident, so the Tesla PR blog blaming the driver had results
proving again how hard it is to undo a lie/almost lie.

~~~
Animats
Also, the people still commenting that the Uber video is meaningful. That's
not from the vehicle sensors. That's from a completely separate dashcam and
recorder.

------
Fricken
A video was released today showing what the classifier in Tesla's Autopilot
sees, provided by an individual who hacked it. False positives galore:

[https://youtu.be/Yyak-U2vPxM](https://youtu.be/Yyak-U2vPxM)

~~~
mephitix
Dunno about ‘galore’. The color of the circles seems to correlate with the
confidence.

~~~
Isinlor
It's stationary (orange) vs. moving (green). Tesla ignores stationary objects
detected by radar and the video makes it painfully obvious why.

------
hartator
> In contrast to the Tesla accident, where the human driver could have saved
> himself from the car’s blindness, the Uber car probably could have braked in
> time to prevent the accident entirely where a human driver couldn’t have.

That's a strong statement. Watching the actual video [1] of the Uber accident,
it seems both an attentive driver (she seems to be not looking at the road
before the crash) or a better decision algorithm, could have prevented the
crash. Keep also in mind, videos are very bad at viewing low light shots, and
the scene must have been significantly clearer to both the driver and the
computer when you're actually there.

[1]
[https://www.youtube.com/watch?v=pO9iRUx5wmM](https://www.youtube.com/watch?v=pO9iRUx5wmM)

~~~
tomp
Why would it be clearer to a compuer? Unless thhe car has some extra sensors
(radar, laser), surely it cannot see better than the camera?

~~~
agency
Uber's self driving cars have both LIDAR and Radar in addition to cameras.

[https://www.gearbrain.com/autonomous-uber-car-
technology-255...](https://www.gearbrain.com/autonomous-uber-car-
technology-2551793303.html)

------
rhacker
Turning one system off in favor of another is basically admitting that they
have no faith in their software. Also isn't driving a car with sensors
(without self-driving tech) basically a giant data collector? The mere act of
driving a car (with a human brain) should help a SDV system understand a
false-positive and a false negative simply by learning what a human does in
contrast to what the machine thinks it is supposed to do.

I imagine this is what separates Waymo and the others. I feel like Waymo is
using math/neural nets, etc.. whereas Tesla and Uber were probably giant hand-
written if/else machines. I have nothing to back that up, except that that's
_kinda_ what this article is about.

~~~
hacker11235
Speaking for Tesla and Uber, they use neural nets. Tesla has the most data
which is what you're referring to above and they do exactly what you were
stating. The software is/has always been shadowing human drivers, that is
learning and comparing to the decisions it would make on its own. Uber is also
most definitely using a similar set up. I think neural nets are a basic
necessity in this field since there are an almost infinite number of
scenarios.

------
ithilglin909
It seems like self-driving cars may not (and maybe should not) become
commonplace until the roads are built for self-driving cars – that might be a
better thing to focus on, instead of systems that mimic human judgment.

~~~
sgslo
Seems unlikely that road infrastructure will be upgraded/changed prior to
construction of self driving cars.

When developing the self driving tech, a company retains all IP associated
with research/development costs. Would they ever instead pay for development
of public infra? Would a municipality build roads that conform to Company A's
standard of self driving roads, omitting the requests of Company B?

~~~
romwell
>Seems unlikely that road infrastructure will be upgraded/changed prior to
construction of self driving cars.

By that logic, railways would have never been built.

"Why would I invest in constructing a special road for that engine of yours?
Make it go on dirt roads as well as a horse does, then we'll talk"

~~~
slededit
Horse and cart never worked all that well, to the point that long distance
travel by land was extremely rare. Deep ruts would form in the dirt roadways
and mud would cause constant obstruction. It was easier sail around South
America than to travel by land between San Francisco and New York City.

It wasn't until the railroad that travel by land was possible at scale. For
the USA and Canada the railroad was seen as the only way to keep the vast land
together as a single nation which is largely why it was subsidized. Further
much of the subdisdy was land grants to land which would otherwise be
economically useless without the railroad to allow transport of goods.

While self driving vehicles may improve efficiency of our roadways it is
unlikely to be anywhere near the same scale of improvement that railroads
brought, and the need to connect the country for political reasons is not
there.

------
bsaul
I think this is one of the first time in my life i see marketing (calling
something « autopilot » which it is not) and overhype (we have to be the first
to have self driving cars, now !) actually kill people and not just rip people
or investor off.

I think history will judge those companies very severely, and i’m wondering if
justice isn’t going to be just as severe right now.

~~~
taneq
Cigarettes were advertised as good for your health, for decades.

------
rasz
This certainly explains reports of almost every Google car accident being the
case of human rear-ending it. Google did it right from the start, didnt gamble
with false positives.

------
Shivetya
I will hold to my opinion stated before, these systems are not safe enough to
be on public roads as of yet let alone in the hands of consumers.

that out I still see no reason why we don't adapt limited access roadways to
support self driving in that realm. a large number of cities have dedicate HOV
and (express) toll lanes that can all be adopted and marked much easier to
support self driving capability in a controlled environment. plus its probably
a good chance than both manufacturers of cars as well as AD systems along with
drivers themselves would be willing to pay more for such access. I just recall
the imagery from the old William Shatner series Tekwar that showed a similar
approach, get on highway and car takes you into the city. It could even then
be extended to service event centers so that it continue driving for you till
parked; think off expressway to airports and stadiums

------
madrox
I wrote about this recently as well. This is actually going to be more of a
concern as we incorporate models into everyday products. Stakeholders and QA
need a solid understanding of Type I and Type II error so they can assess how
much risk they're willing to take on and make that part of the quality
process.

Shameless plug: [https://blog.d8a.me/the-qa-of-stochastic-
processes-a15a94065...](https://blog.d8a.me/the-qa-of-stochastic-
processes-a15a9406519c)

------
fcolas
Nice article; thanks.

* * *

. Tesla claims: A 40% crash rate reduction with the autopilot as compared to
no autopilot, over an 18 month period [1].

. If this is true—and we could imagine it is (at least partially?)—then Elon's
remark to journalists would make sense:

"It's [..] irresponsible [..] to write an article that [..] lead people to
believe that autonomy is less safe,” [..] “Because people might actually turn
it off, and then die" [1]

* * *

But to have an opinion about the autopilot's risk statistics I would also need
to know: a) What populations (data) they compare; b) How each population is
defined (inclusion and exclusion criteria); c) What's the sample size (18
months, and?); d) Who makes these calculations (to clearly identify possible
conflicts of interests).

\- Not sure if this type of data is publicly available?

* * *

Actually the National Highway Traffic Safety Administration (NHTSA) seems to
indicate[1]: 1) that the data comes from Tesla—cf. point d) => conflict of
interest; 2) Autopilot on/off was NOT used for the risk statistics—although
it's central (point a); 3) instead the "40%" would measure the "number of
airbag deployments per million miles" which is a proxy-metric that's not
directly related to car accidents.

\- Hey, this is odd (it's definitively not a Science or Nature method
protocol).

* * *

. "The Insurance Institute for Highway Safety suggests: A "13%" reduction in
collision claim frequency, indicating sedans with Autopilot enabled got into
fewer crashes that resulted in collision claims to insurers."

However it's a small difference and there're possible confounders like social
status (a "Tesla driver"), gender, geographical area, and usually the
confounders have a large influence on experiments, so it's unlikely that this
(small) 13% difference would remain if we adjust for confounders...

* * *

Here's the article AARIAN MARSHALL on Wired. [1]
[https://www.wired.com/story/tesla-autopilot-safety-
statistic...](https://www.wired.com/story/tesla-autopilot-safety-statistics/)

------
LinuxBender
Could any of the fatalities be related to the malicious code commits? [1]

[1] - [https://www.fastcompany.com/40586864/read-elon-musks-
email-a...](https://www.fastcompany.com/40586864/read-elon-musks-email-
alleging-there-is-a-saboteur-at-tesla)

------
dsfyu404ed
>It may seem cold to couch such life-and-death decisions in terms of pure
statistics, but the fact is that there is an unavoidable design tradeoff
between...

The fact that we as a society can't have an adult discussion about that topic
is a large part of why it's so hard to strike the right balance.

------
tzakrajs
Good thing L5 self-driving can be done without low-latency LIDAR and that
Tesla didn't defraud many Model S owners by selling them cars fundamentally
incapable of providing L5 self-driving.

/s

------
sgslo
The upside of self driving vehicles is so immense that I can't help but find
in favor of giving leeway to companies developing this tech. I would vote
legislation limiting the accident liability of 'qualified' companies
developing self driving tech. No, I don't know what separates 'qualified' from
'not qualified'.

Try looking at this a different way: there were 37,461 vehicle related
fatalities in 2016 in the US alone. In the current risk-averse climate, would
we - over the early 1900's - ever allowed the development of public
infrastructure to support an invention leading to so many fatalities? Likely
not, but the benefit we enjoy from motorized transportation far outweighs the
cost of those 37k yearly lives. The point is that continuing with a risk-
averse, liability-obsessed development culture will stifle invention that can
otherwise lead to great quality of life improvements.

~~~
Semirhage
Then you and other people like you can volunteer to work at a test track as
obstacles. Leave the rest of us out of it. You’re entitled to your utopian
fantasy of what _might_ be, and the best way to express that is to put your
life on the line instead of volunteering others to do it.

 _Statements like this seem to be trying to make the point that any amount of
increased public risk is unacceptable unless everyone opts in to accept it._

No, statements like mine reject the notion that sacrificing lives today for
the possibility of an unproven improvement sat some unspecified later date is
morally reprehensible. My answer is to invite the person endorsing this to put
their own life on the line instead. If we were talking about medical testing
on humans instead of cars, maybe it would be easier for some to understand.
“It’s so bad today, the only course of action is to act recklessly to improve
things as fast as possible!” is a thoroughly rejected argument.

~~~
zaroth
Statements like this seem to be trying to make the point that any amount of
increased public risk is unacceptable unless everyone opts in to accept it.

I find this argument entirely uncompelling. There are an inordinate amount of
risks imposed upon me while operating in society which I neither agreed to nor
would I given the choice, aside from in a hand-wavy “social contract” sense.

However as a member of society I am in fact compelled to accept myriad risks
on a daily basis imposed on me for the benefit of others, or the benefit of
potential scientific advancement.

This concept that a society cannot morally make risk-reward decisions in the
course of technological advancement is entire bunk IMO. Government can
absolutely morally make these trade offs and does so _constantly_.

I feel pretty good about the current legal and regulatory frameworks and the
technical chops of the NTSB to monitor developments in self-driving R&D
adequately.

One thing I am certain of is that we _absolutely must_ succeed in developing,
productizing, deploying, and ultimately mandating self-driving technology.
Millions of lives and trillions of dollars are at stake, and yes, society as a
whole will incur individual loss of life in the pursuit of this goal. An
attempt to develop the technology with zero risk of collateral damage would in
fact cost many more lives overall.

~~~
dbt00
So far, per mile driven, the fatality rates of Uber and Tesla SDVs are far
higher than human drivers in statistically safer than average vehicles.

Waymo has a better track record but essentially refuses to drive at
significant speed -- I have yet to see a Waymo SDV place itself in a situation
with a higher speed limit than 35mph, and I see them a lot.

~~~
mclehman
I regularly see the Waymo vehicles in 45 mph zones on my commute. I can't say
I've seen them go any faster, but I spend about 95% of my time on roads marked
45 mph or less.

