
New way to test self-driving cars could cut 99.9 percent of validation costs - linuxbox_gmail
https://phys.org/news/2017-05-self-driving-cars-percent-validation.html
======
shmageggy
> _the new accelerated evaluation process breaks down difficult real-world
> driving situations into components that can be tested or simulated
> repeatedly, exposing automated vehicles to a condensed set of the most
> challenging driving situations_

Chess engine programmers do something like this. Rather than playing thousands
of full games to determine your engines strength, you can get a pretty good
snapshot by running custom test-suites that contain a curated selection of
known challenging positions [1]. One well known downside to this is that if
you optimize your engine for test-suites, it invariably performs worse in
overall general gameplay. Maybe we'll end up in a situation where cars learn
to drive in an extremely over-cautious and risk-averse manner in order to pass
all of the dangerous test scenarios.

[1] [https://chessprogramming.wikispaces.com/Test-
Positions#Test](https://chessprogramming.wikispaces.com/Test-Positions#Test)
Suites

~~~
zeta0134
Perhaps I'm pulling teeth here, but at 70+ MPH, I don't think there's any such
thing as overly-cautious until it backfires and becomes less safe in the
process.

~~~
derefr
I think the "overly cautious" bit would be never thinking it's safe to come up
to 70MPH in the first place.

~~~
gumby
Indeed, one thing I look forward to is SDCs returning residential streets to
their original primary users: humans.

~~~
closeparen
Self driving cars will actually take us away from a world where all car trips
are humans accomplishing their goals, to a world where a decent proportion of
car trips are machines serving their own purposes.

As an aside: please do lean further into this "people whose transportation
needs are best served by cars aren't humans" rhetoric. It's going to be a lot
easier to destroy at the polls.

I say this having just sold my car to move to a tiny studio in an urban high
rise near work, because I recognize that my top-10% salary and lack of
children are the only reason I can do that. I'm excited by the possibility of
self-driving cars to bring the benefits of the automobile to more people, and
most excited about those it can liberate from slow, crowded buses.

~~~
gumby
As derefr said I meant use of the roads for walking, playing, sitting, etc
like they were before the "motor car" became common. The SDCs will have to be
more careful.

Roads are now intended for cars, and new infractions (like jaywalking) were
invented to give automotive use primacy over others. Cars should merely be one
use of the roadways.

Not sure what you mean by "It's going to be a lot easier to destroy at the
polls."

~~~
closeparen
"Cars should merely be one use of the roadways" is equivalent to "people who
can't afford decent urban homes should live in shitty urban homes." The
primacy of the car is what lets us currently live in decent suburban homes.

Urban and suburban residents have competing interests here: urban residents
(for whom cars are a nuisance) would like to safely frolic in the streets.
Suburban residents (whose lives depend on fast travel by car) would like to
continue affording decent homes.

This competition will appear in city-level politics in various ways. The anti-
car side is handing the pro-car side some ammunition when talking about them
as subhuman (i.e. framing drivers and people and mutually exclusive
categories).

------
josephcole
Hi admin I examined your post about Facecord / Bushcord measuring Firewood. It
was quite an impressive blog and it carries so much important and useful
information, and I also liked the way you presented the blog i was amazing.
Thanks for sharing!
[http://www.juegosdefriv.live/](http://www.juegosdefriv.live/)

------
JoeAltmaier
Lets pass on the issue of memorizing the training data. The article also says
the test is designed

    
    
       "to consider human drivers the major threat"
    

I'm thinking that degenerate driverless-algorithm behaviors will be even more
problematic. Imaging an imminent head-on collision between two driverless
cars. They could play 'dance with me', each dodging to the right or left only
to see the other car do the same. In a few milliseconds they could lose the
opportunity to avoid the accident.

With a human driver and their limited response time, the other car is actually
_more predictable_ and interactions easier to manage.

~~~
smilekzs
You can make a rule, just like ships already are doing:

> When two power vessels are approaching head on,both vessels should alter
> course to starboard to pass port-side to port-side.

> When two power-driven vessels are in crossing situation on a collision
> course, give way to the vessel to starboard (right).The give way vessel must
> take early and obvious action to avoid a collision by either stopping or
> altering course to starboard.

[https://maritime.college/Boating-Rules.php](https://maritime.college/Boating-
Rules.php)

\---

Plus, driverless cars can potentially use more efficient short-range radio and
figure out themselves.

------
onion2k
_it would take nearly a decade of round-the-clock testing to reach just 2
million miles in typical urban conditions._

...if you only use one car.

~~~
devinl
As stated in the article, they need to reach 11 billion miles to get the
confidence level they are looking for. So extrapolating this, it would take
5500 cars a decade of round the clock testing to actually meet the guarantees
they need (90% safer than human driven at a 80% confidence level). You still
could do it in parallel; it'd just require a lot of test cars.

One part that is unclear is that the 11 billion mile requirement doesn't
specify urban conditions and the 2 million mile per decade estimate does. So
there might be some errors there.

------
patall
And then, car makers will try to detect these tests and make their cars more
cautios if in such a situation. Its probably going to be great for training
but testing should be run in the real world, could finally also reintroduce
realistic benchmarks for mpg and other running costs.

------
11thEarlOfMar
There will always be unanticipated circumstances that will foil the sensing
systems. My go-to example is this situation where road re-paving artifacts and
shadows combine to create a false lane marker. This vehicle's video-based
system determined that the bright area between the shadow and lane pavement
was the left lane marker and sounded a lane departure alert. Had the
autonomous lane following feature been activate, it would have reacted and
attempted to center the car on the false boundary:

[http://imgur.com/a/KnsXU](http://imgur.com/a/KnsXU)

------
kefka
Hmm.. Randomly generated real-world traffic based from traffic sensors and
cameras. The key here is, if they use the same data and feed every car the
same, then it can be easily gamed with overfit techniques.

The better question is, what happens if you give the AI car a no-win
situation? How does it handle it? The Kobayashi Maru can be looked at as a way
to detect outliers and failure modes... even if the game is rigged.

------
mturmon
Unfortunately, the linked white paper is _very_ thin on methods and analysis.
It, itself, is not peer-reviewed, and it points to only one peer-reviewed
publication to justify the approach.

------
sharemywin
seems like a good dataset for training a deep learning system.

~~~
frgtpsswrdlame
This needs to be an out-of-sample test.

