
Uber Neglected Simulation Testing on Self-Driving Cars, Insiders Say - whatok
https://www.theinformation.com/articles/uber-neglected-simulation-testing-on-self-driving-cars-insiders-say
======
rasmi
For comparison, here is a look at Waymo's simulation operations:
[https://www.theatlantic.com/technology/archive/2017/08/insid...](https://www.theatlantic.com/technology/archive/2017/08/inside-
waymos-secret-testing-and-simulation-facilities/537648/)

~~~
dclowd9901
> Being in the car a lot, I can feel what the car is doing—it sounds weird,
> but—with my butt

He's describing, incidentally, how professional drivers drive.

~~~
hangonhn
Can you expand on that a bit more?

~~~
Tuxer
As you become "good" at driving in harder conditions, you'll use your butt
(which is really how you're physically connected to the vehicle) to figure out
the traction state of the car, to know if the car is sliding, if it's close to
losing the rear end or the front end, if it's pitching down or up, and so on.
This is how you know which corrections to make to the vehicle to keep it going
fast without losing control.

Pilots of small planes do the same thing. After a while you can just feel
small accelerations and change of directions without needing to check some of
the instruments.

~~~
QuercusMax
Huh, fascinating. I remember reading a Star Wars novel ages ago in which they
said that one of the things that made Luke Skywalker a good pilot was that he
slightly detuned his craft's inertial compensators (or whatever the Star Wars
version is) so he could feel how the craft responded. Didn't realize that was
actually based on real life driving/flying!

~~~
bmelton
The common complaint against "new" vehicles and their drive-by-wire systems is
that they functionally act as inertial compensators, dampening road feel.

If the roads were perfect, you wouldn't necessarily want them (day to day),
because it diminishes the minute vibrations that make their way up to the
pedals, into the steering wheel, etc., and those things give you feedback into
what the car is doing that are invaluable.

Of course, the roads aren't perfect, and most of us aren't doing high-
performance driving on them, and we don't need anywhere near that degree of
feel, so the benefits of driving by wire far outweigh the cons of analog
driving.

------
qasar
Qasar here (x-YC founder and partner). So im building a simulation company.
Not commenting on Uber specifically but the reason lots of companies don't
have great efforts is simulation is actually really hard. It’s not building a
web application (tested methodologies, established best practices, lots of
experienced engineers in the space...etc).

You can get a lot done (and get wonderful demos) through research vehicles.
The issue with sim is similar to the issues with AVs as a whole - getting to
90% is much easier than getting to 99.9999%.

Also, there is this weird thing in AV (and specifically in-house sim) where
pointy-haired managers think "if we put 30 SWEs on the problem - we'll make
progress!". While raw eng horsepower matters in some software problems, when
you are on the line between research and engineering, more SWEs might actually
make getting a robust product up harder.

Another thing we've seen is in-house efforts overfit solutions. They don't
really have a wide viewpoint (nature of how secretive the space is), and
therefore go down paths which are just plain wrong but no one is there to
challenge them.

~~~
solatic
Is this really so surprising?

Nobody (well, theoretically someone, I'd like to be proven wrong) has a self-
driving algorithm which is formally proven. If we can't get, to use your
example, web engineers to formally prove their web applications, when they
have all these structural advantages, then how can we expect something
unproven (har, har) to be formally proven? So there needs to be an extensive
QA test suite.

And for all of these other relatively-mature engineering sectors like web
engineering, how many of these companies are really investing in QA? In strong
and comprehensive test suites, in security reviews, in performance testing and
UX scoring before release? Almost nobody, right? I mean, the only reason why
society can get civil and biomedical engineers to submit to strict quality
control is through regulation, but no regulatory body has any real idea of how
to apply that to software. And autonomous driving systems are supposed to be
different?

Mythical Man-Month came out _38 years ago_. The Pareto Principle, not so much
as being explicitly a 80/20 split but about how the vast majority of the
effort is needed to achieve actual completion rather than just a proof of
concept, was originally coined _in 1896_. Suddenly it doesn't apply to Uber?

Is it really so surprising that it's so difficult to find senior practioners
in these areas when few companies are committed to that level of quality, and
the companies which are tend to be so successful that they can afford to pay
compensation at rates so far above the rest of the market so as to ensure that
few of their senior engineers ever leave?

~~~
woolvalley
The reason why there isn't perfect QA for things like webapps is a cost
reason, not a knowledge reason.

Society usually doesn't need a formally proven cat pics sharing app, and most
web apps, including facebook, are that level of application. They would rather
pay a cheaper price than to have a perfect expensive thing.

You'll also notice that as organizations get bigger, they do invest more in
testing, QA and regression detection.

~~~
speedplane
A counter example is in the aerospace industry. Most software has to be d-179b
certified, which requires not only testing with complete code coverage, but
also complete branch coverage. Every possible outcome in every “if” statement
must be tested.

Needless to say, this is expensive. I’ve heard estimates that it costs roughly
$1000/line of code.

The point is that there are methods to increase reliability, but there isn’t
the financial incentive to do so.

------
beat
Given time to market constraints and the potential value of having widely
adopted autonomous vehicle software, it's not all surprising that vendors will
take experimental shortcuts. And it's not at all surprising that sometimes,
those shortcuts will result in fatalities.

That being said, I still want this tech on the road ASAP, not because it's
perfect, but because human drivers are So. Fucking. Awful. They kill thousands
of people a year due to negligence, incompetence, and occasional malice. Plus,
the additional sensors available and the software response time mean
autonomous drivers can outperform even sober, attentive, skilled human drivers
in most situations.

A couple of weeks ago, I saw someone talking scary stuff about what someone
here called "semi-autonomous murderbots" in a FB post that appeared on my feed
immediately after several posts about a local accident, where a bicyclist was
killed by a road-raging driver, who then hid/destroyed his car (the murder
weapon still has not been found, even though his trial is now over). If that
jackass had a self-driving vehicle, the bicyclist would still be alive today.

So are big corporations going to do kind of a careless job testing live
vehicles in a rush to market? Yes. Will people get hurt? Yes. Is it worth it?
Yes.

~~~
ksk
>So are big corporations going to do kind of a careless job testing live
vehicles in a rush to market? Yes. Will people get hurt? Yes. Is it worth it?
Yes.

Hey, another human driver on the freeway here. Sorry, but I don't want to be
killed because you want to play with some toys. It is completely unethical and
immoral to ship unproven tech that has the potential to kill people.

I'd suggest running an experiment with OPT-IN volunteers in a neighborhood or
small city for a year or however long it takes and then coming back with data.

~~~
beat
It's not about "play with some toys". It's about _saving lives_. Tens of
thousands of people are killed by cars every year in the US alone. And the
primary causes of those fatalities are the kinds of things most easily
addressed by automation - lane drift. Running red lights and stop signs.
Speeding. Driving too fast for conditions. Aggressive maneuvering leading to
loss of control.

Drunken or distracted driving, by itself, doesn't kill. It leads to the
behaviors that kill, like lane drift and speeding and running lights. These
are all things that autonomous cars handle really well.

~~~
sincerely
When a human-driven car kills someone, the driver is held responsible and
faces some sort of punishment. When a self-driving car kills someone, it's not
clear that there are consequences, meaningful or otherwise, for anyone besides
the person killed. Until this changes, I am fine with having a hardline stance
against autonomous vehicles on the same road I'm on.

~~~
doikor
Volvo has the right approach. If the fault is on their car when it is in
autonomous mode they will take full responsibility. They have said so multiple
times publicly. Then again they don’t trust their system enough to give it to
actual customers yet.

~~~
w_t_payne
My experience with Volvo was very positive. They were very keen on building a
Simulink model for the entire vehicle, which IMHO is a good way to go.

------
srcmap
It should be mandatory for all self-driving car (system/software) vendors to
publish their test plans and all test logs for public to review before the
car/software are allowed to be on the public road.

It likely allows everyone to learn from all the mistakes and improve the
process for the whole industry - similar to the passenger airplane safety
record improvements learned from every single accidents/crashes.

~~~
blevin
Perhaps we will end up with a standardized validation simulation model that
autonomous vehicles must pass for licensing. The test suite could grow as more
anomalous scenarios are analyzed (ex: acceleration into gore point; white-
painted trailer truck turning across divided highway; debris on highway), and
it would ensure that any new vehicle on the market or OTA update would meet a
minimum of safety. Manufacturers could compete on test suite coverage and
acceleration curve stats, and market using rendered video of simulation
results showing superhuman skill.

~~~
w_t_payne
I hope there is some level of cooperation.

At the very least we should have a standard set of crash or near-miss
scenarios so that we can collectively learn lessons when things go wrong.

Particularly when people have already paid the price with their lives.

------
duncanawoods
I don't understand why Google/Uber are live testing with lethal SUVs instead
of polystyrene golf-carts that could only hurt someone if they tried to eat
it. We could be getting acclimatised to near disposable pizza delivery carts
earning the algorithm enough trust for 2 tons of metal and a passenger.

~~~
wffurr
Heck I don't understand why most people drive lethal SUVs for short trips that
could be much more efficiently accomplished with a golf cart aka NEV.
Especially in dense urban areas where your average speed ends up being only
10-20 mph anyway.

~~~
TeMPOraL
\- People want big cars and big cars are cheap enough for them to get.

\- For the perceived feeling of safety in larger cars.

\- For the off chance that once a year they might need to take some extra
large load (e.g. help a friend with moving).

AKA. emotional needs, not practical ones.

~~~
ectospheno
It snows where I live. My street floods. I have a family. I frequently want to
carry a large number of items.

I get that SUV's aren't for everyone. Acting like they are for no one is silly
though.

~~~
mikestew
_It snows where I live._

I grew up where it snowed a lot. I've never owned anything that sat
particularly high nor with 4WD, and I've never missed a day of work due to
snow.

 _My street floods._

If the water is above the air intake of a Honda Civic, your SUV isn't going to
do you any good.

 _I frequently want to carry a large number of items._

I'm continually shocked at how few SUVs exceed the cargo volume of our 35mpg
Scion xB. I've moved furniture for a friend because his Equinox wouldn't carry
it. I rented a Nissan Xterra to carry a bicycle box that fit less snugly in
the Scion than it did in the Nissan. SUVs might have a use, but carrying
things is not one of them.

Until you threw in the ol' "I have a family" (because the children can _not_
be seen in a Volvo wagon), I thought you were doing a parody of "Excuses SUV
Owners Use to Justify Their Big-Ass Vehicle". I'm pretty sure I was wrong.

~~~
dsfyu404ed
>I'm continually shocked at how few SUVs exceed the cargo volume of our 35mpg
Scion xB. I've moved furniture for a friend because his Equinox wouldn't carry
it. I rented a Nissan Xterra to carry a bicycle box that fit less snugly in
the Scion than it did in the Nissan. SUVs might have a use, but carrying
things is not one of them.

That's like saying you don't need a fullsize truck because the methheads seem
to do just find hauling scrap metal in their Dodge Caravan.

For a load with a constant amount of space usage (e.g. kids + sports stuff)
having more space will make hauling that load less of a chore. If you do it
regularly then the luxury of a bigger vehicle is often considered worth the
cost.

I can and have hauled fridges, office desks, etc on the roof of my wagon. I
always prefer to use my truck for those things unless the item can't stand
getting dirty from the truck or I'm going so far that the extra fuel economy
and better driving experience of the car makes it worth it. Strapping a desk
to the roof of a car and not doing a crap job is a chore. Just tossing it in a
truck and strapping it down takes like 30sec.

A vehicle with more space can haul more crap or haul the same crap more
easily.

As an aside, I wouldn't let my kids be seen in a Volvo Wagon. I'm white trash,
not some hipster. A rusted out Subaru is more my style.

~~~
mikestew
_That 's like saying you don't need a fullsize truck because the methheads
seem to do just find hauling scrap metal in their Dodge Caravan._

That's like saying that one kinda, umm, perhaps didn't completely comprehend
my point. Unless you're arguing that a Dodge Caravan has more space than a
Ford F-150 (and maybe it does, dunno).

 _having more space will make hauling that load less of a chore._

Yes, I completely agree, that's why I like the Scion over an SUV.

 _As an aside, I wouldn 't let my kids be seen in a Volvo Wagon._

So you're worried about the image you project as well. Fair enough, I think
that might apply to many vehicle owners including, to a degree larger than I
care to admit, myself.

------
refurb
Not sure if this is possible, but couldn't they capture all the LIDAR inputs
from a car being driven by a person, then feed that into the software to see
if it actually recognizing objects correctly?

Or even do it on a closed track. That way you could throw all sorts of things
at it like kids, bikes, garbage cans, whatever.

Should be able to see what works and what doesn't work pretty quickly.

EDIT: Never mind! That's what Waymo does in the article another person posted.

[https://www.theatlantic.com/technology/archive/2017/08/insid...](https://www.theatlantic.com/technology/archive/2017/08/inside-
waymos-secret-testing-and-simulation-facilities/537648/)

~~~
Klathmon
When it comes to "closed track" and other "simulation" style things, you need
to be very careful when ML is involved.

I don't know if Uber/Waymo/Others have "solved" this problem, but ML systems
have a tendency to "learn" the wrong thing. You could run millions of
simulations (both virtual or on a closed course) with "test obstacles", but
the second the car encounters one a child with red balloons in the early
morning on a foggy day it might not know what to do with it because that exact
scenario wasn't in any simulations.

~~~
caboteria
They'd be using the simulator for validation, not training.

~~~
refurb
Exactly. Throw something at it the system should stop for (e.g. a child). If
it doesn't, go back to the drawing board and figure out why.

Agreed the test track is not ideal, that's why I was wondering if they could
throw real recorded data at it. No risk to the public, but puts the system
through real-world trials.

------
kerng
Testing in Production. It still boggles my mind that that's the norm in
software engineering these days. Interesting how in this case it seems
"inappropriate" in real life, and the article fails to make the connection how
normal this is in the software world nowadays unfortunately.

------
hannasanarion
The Uber crash was not a failure of the AI, it was a failure of the UX.

The car saw the obstacle and detected a collision, but there was no warning
system to alert the driver about it.

Meanwhile, the driver was looking at a laptop and taking notes on the system
logs (which they were told to do), not watching the road.

~~~
cozzyd
The car saw the obstacle and detected a collision, but did nothing, presumably
due to the high false positive rate of such things.

~~~
aetherson
Yeah, I feel like your parent poster is ignoring the implications of the high
false positive rate.

Like, let's say I had a grammar check. But the grammar check was _bad_. It
flags every other sentence as needing a grammar change, anything other than a
very simple "subject-verb-object" sentence structure looks wrong to it. For
every correct grammar problem it identifies, it finds 20 incorrect grammar
problems.

As a result, I turn off the UI for the grammar check.

Then I write a document that has a grammar problem.

It seems stupid to say that this is either a failure of my use of the
software, or a UI failure of the software. The problem is that the _fucking
software doesn 't work_, specifically that it fails in a way that makes it too
"noisy" to use.

And sure, if my job is to catch grammar errors, then it's also my fault. I'm
supposed to be backing up the system. But we also know that I won't be perfect
and will have failures of attention sometimes. But regardless of that, the
grammar checker just didn't work.

And, in this analogy, bad grammar kills people.

~~~
voidmain
In this case, as I understand it, they also disabled the emergency braking
feature built into the Volvo. Which wouldn't be obviously crazy if they had
better, adequately tested emergency braking integrated with the "autonomy"
system. But to disable the existing safety feature in favor of their own
system, and _also_ disable their own system due to a high false positive rate,
seems negligent on the face of it.

------
pmarreck
Of all the things that managers neglect to ensure rigorous testing of, self-
driving cars should be the worst possible offender

~~~
w_t_payne
I would have hoped that ISO 26262 would help to prevent this sort of thing
from happening. Maybe it is an audit quality issue?

~~~
pmarreck
There is also likely difficulty in testing an ML black-box. When I took the ML
class (Stanford's, online), they actually recommended setting aside some
percentage of the training data as test data, but this is (IMHO) sub-par
because it doesn't eliminate the possibility of things like overfitting

------
rossdavidh
The right question is not "why did Uber do this?", but rather, "what was it
you thought would stop them from doing this?" Current answer: nothing. But
hopefully, the attitude of governments towards letting a company like Uber
test fast-moving heavy machinery on their citizenry, is changing. Different
industries have different attitudes towards innovation and risk, and there are
good reasons for that difference.

~~~
w_t_payne
I think that the automotive industry could take a leaf from the aerospace
industry.

A lot of aerospace money has been spent learning how to do things safely.

It would be a shame to spend the same money all over again, re-learning the
exact same lessons in automotive.

~~~
wilsonnb2
Most of the automotive industry already knows how to do things safely. It's
really just companies like Uber and Tesla that either don't know or don't
care.

~~~
w_t_payne
Aerospace and defence seem to have a much stronger notion of systems
engineering than I experienced in Automotive -- but it is entirely possible
that what I experienced was an outlier, not representative of the whole.

------
crooked-v
"Move fast and kill people" is the new motto to use, I guess.

------
ConceitedCode
I never understood why the starting point for these projects was not simply to
drive a car in GTA V (or some other game) around the city. It would test
pedestrian detection, street signs, traffic lights, merging, emergency
vehicles, etc... While certainly not perfect, it would seems like a good
starting point in a controlled environment that is already modeled on the real
world.

~~~
glup
Isn't it? I was under the impression that this was the first step was
simulators like CARLA ([http://carla.org/](http://carla.org/)) before real-
world testing.

~~~
ConceitedCode
Thanks for the link! That is a fascinating project.

I was not considering the different sensor inputs that would not work on an
existing game.

------
cozzyd
I guess this is both astonishing and completely unsurprising at the same time.
How can one hope to gather decent statistics in exceptional situations without
a simulation?

~~~
w_t_payne
A big problem is that the software engineers and algorithms engineers are
pressed to deliver demonstrations in very short timescales.

This precludes the necessary conversations about test interfaces, for example,
making the task of the simulation team closer to one of reverse-engineering
than of simulator integration.

~~~
IncRnd
And people getting run over is the result of those software decisions by
management, software engineers, and alg. engineers. Don't just be a cog in the
machine, doing what one is told to do. Write software in the ways that are
known to be correct and responsible.

~~~
_jal
This is important. I hope engineers working autonomous cars think long and
hard about whether they want to be on a team that, it looks to me like,
negligently killed someone by choice.

The only difference between disabling the emergency braking mechanism and,
say, intentionally disabling safeties on industrial machinery is the novelty.
I think it was pretty obvious that something like this would happen, and
that's why so many people have been making noise about it. Now that we know
for a fact that Uber is willing to make this choice, there's (a) no excuse for
not coming down on them for it, (b) no excuse for Uber not completely
overhauling their safety protocols with the help of an external ethics and
legal advisors, and (c) no excuse for engineers working on this for not at
least seriously thinking about their own ethics and possible moral
culpability.

Some people want to focus on the future. Self driving cars will be great!
Gotta break some eggs to make omelets! Fine. But those who are killed before
that bright future deserve some accountability, those who don't want to be
killed deserve some respect, and the makers of said omelets can bloody well be
expected to be held responsible for their actions.

------
cabalamat
Move fast and break people.

------
locacorten
In my opinion, this is criminal behavior and Uber's senior leadership should
go to jail.

~~~
wpietri
100% agreed. Somehow large corporations get away with a diffusion of
responsibility. Enough people are involved that it's hard to blame one person.
But certainly for issues resulting in death, I don't think responsibility
should be diluted.

If 10 different executives knew or should have known that their decision had
life-critical implications, then I think all of them should face the
appropriate penalty. (Here, I presume it's for involuntary manslaughter due to
criminal negligence.) And I think the same penalty should apply to developers
who knew or should have known that their actions put lives at risk.

