
Upgrading Autopilot: Seeing the World in Radar - _miqu
https://www.tesla.com/blog/upgrading-autopilot-seeing-world-radar
======
Animats
More info: Tesla's radar is a Bosch device. Either the Bosch LRR4, which is
several years old, or the Bosch mid-range radar. Bosch has made about 10
million of these and related models. Tesla is not the only customer.

This device isn't enough for a full point cloud. It doesn't scan in elevation,
just azimuth. Some variants do have an upward-pointing beam in additional to
the main forward beam, which is about 5 degrees in vertical.

There are automotive radars which scan in 3D[2], but Tesla's is not one of
them.

Small radars are rather blunt instruments. You tend to get one point for each
target, not lots of points. The beam focus isn't that tight. Tighter focus
requires a larger antenna array.

[1] [http://www.automotiveworld.com/news-releases/bosch-
presents-...](http://www.automotiveworld.com/news-releases/bosch-presents-new-
radar-sensor/) [2] [http://www.fujitsu-
ten.com/business/technicaljournal/pdf/38-...](http://www.fujitsu-
ten.com/business/technicaljournal/pdf/38-1.pdf)

~~~
omarforgotpwd
I believe you, but I think that misses the point. My takeaway here is that
Tesla is just getting started with Autopilot.

Despite the hype, this blog post makes it clear that Autopilot is just a very
simple camera based system that requires human monitoring (but is still very
useful in my personal experience). Now they are releasing a software update
that uses the previously unused radar hardware as well as machine learning to
achieve better results. This is exciting because they're going to keep
releasing software updates over the air and the software is just going to keep
getting better and better.

The fact that the hardware is shitty doesn't concern me. Google's self driving
car project depends on a Lidar system that costs $75,000 today. That's almost
four times what the average car costs, and it doesn't work in bad weather.
Regardless of how good Google's software is it could mean nothing if Tesla
acheives a similar or better result using cheap commodity hardware. The
hardware will only get better over time at the same price.

Tesla is in a better position to bring full autonomy to market than anyone
else, since they control the hardware, the software, have cars in the field
etc. For this reason I wouldn't be surprised if Tesla becomes the first
company to break a market capitalization of $1 trillion dollars. Computers
gaining the ability to move around the world with drones and autononomous
driving will have an economic impact bigger than the introduction of the
internet. Where we are now is meaningless. What matters is that Musk has
stated the goal and we have something in the field today that can be updated
iteratively over time until it's perfect (which will be never). We may not be
very far along this journey, but we've taken those hardest first steps towards
the next big technological revolution that will once again change _everything_
about how humans live, the implications of which we can't even begin to
imagine.

~~~
Animats
Volvo is in a much better position. They're going live with 100 human drivers
next year, actual customers, not employees, with drivers _not_ required to
have their hands on the wheel.[1] This will be enabled only for well-mapped
roads in Gothenburg, but they're not all freeways. Volvo's CEO has publicly
taken the position that if there's a crash in auto mode, it's Volvo's fault.

Volvo uses four cameras (one of which is a trifocal 3D unit), four radars, one
LIDAR, and 12 ultrasonic sensors. That's a reasonable sensor suite for this.
Tesla has one camera, one radar, and some ultrasonic sensors. Not enough.

Volvo is also way ahead on self-driving car commercials.[2]

[1] [http://www.volvocars.com/intl/about/our-innovation-
brands/in...](http://www.volvocars.com/intl/about/our-innovation-
brands/intellisafe/intellisafe-autopilot/this-is-autopilot) [2]
[https://www.youtube.com/watch?v=bJwKuWz_lkE](https://www.youtube.com/watch?v=bJwKuWz_lkE)

~~~
jaimex2
Volvo and others can say whatever they like, until they bring something to
market I will assume they are freaking out as how they will slowly be made
irrelevant.

~~~
imtringued
The companies that are being made irrelevant are usually "stuborn" and have a
strong belief that their current busines model and products will prevail.
Kodak and nokia refused to adopt modern technology until it was too late. I
don't think developing autonomous cars counts as sticking with the status quo.
They may be behind google or another company but they are not wasting time
twiddling their thumbs.

~~~
tim333
I'm not sure that applies to Kodak or Nokia. They saw the changes coming,
Kodak launched digital cameras, Nokia tried launching MeGo. The trouble is
they didn't have a competitive advantage over all the other digital camera
companies or phone OSs.

By the way I don't think Volvo is freaking out so much as going for it's
vision 2020 thing:

"in 2008 we set out our vision that by 2020 nobody should be seriously injured
or killed in a new Volvo car."

------
Vik1ng
> "The update will also penalize inattentive drivers. If the car determines
> that the driver doesn't have their hands on the wheel and throws its audible
> warning three times in an hour, it will lock the driver out of the feature.
> In order to re-enable Autopilot, the car will have to be pulled over and put
> in park."

[https://www.engadget.com/2016/09/11/tesla-s-next-
autopilot-u...](https://www.engadget.com/2016/09/11/tesla-s-next-autopilot-
update-will-rely-more-on-radar/)

If Engadget got that right I think we will see a lot of upset Tesla owners in
a few weeks.

~~~
Animats
There will be more upset Tesla owners, but fewer tuned-out ones. Tesla needed
to do this.

~~~
toomuchtodo
Tesla punishing users not following the rules > Regulators punishing Tesla.

------
karyon
Is it just me or does whitelisting static objects to determine whether the car
will collide with them seem like a bit of a crude hack? It almost sounds like
the system will brake at newly installed traffic signs.

edit: Upon closer reading, he explains it somewhat. Once they have enough
data, the system will start braking on unknown objects with gradually
increasing force as the confidence level rises. So basically, it will brake on
unknown traffic signs but only slightly, as the confidence level shouldn't get
too high, if I understand that correctly.

The last paragraph sounds technically challenging and interesting:

"Taking this one step further, a Tesla will also be able to bounce the radar
signal under a vehicle in front - using the radar pulse signature and photon
time of flight to distinguish the signal - and still brake even when trailing
a car that is opaque to both vision and radar. The car in front might hit the
UFO in dense fog, but the Tesla will not."

edit: it seems like they are already doing it beginning with this update: "Now
controls for two cars ahead using radar echo, improving cut-out response and
reaction time to otherwise-invisible heavy braking events". that sounds
awesome.

~~~
MarcelGerber
Yes, this seems really complex to me, too. I'm also not quite sure on why
exactly it is so complex to distinguish between objects (including vehicles)
on the road and ones above/next to it. If they have radar images (I imagine
them as images with depth information, which might be fundamentally wrong),
they should be able to tell both where the road is going and, with that
information, which of these objects are of relevance.

But for the learning part, they probably combine the camera that they used to
date as their primary device in combination with the radar (at least in
daylight scenarios) to identify objects. They may even be able to learn about
the special material properties, like the reflective coating of a traffic
sign.

~~~
toomuchtodo
> I'm also not quite sure on why exactly it is so complex to distinguish
> between objects (including vehicles) on the road and ones above/next to it.

Because radar does not have the same resolution as LIDAR.

EDIT: Phased array radar and cheap stationary LIDAR should get under $100 in
~5 years, at which point this whole argument will be moot. Hacks in the
meantime!

~~~
Pyxl101
What does the world look like in radar? Are there any visualizations
available?

Could multiple radar emitters and receivers be used to create a phased array
and improve resolution?

~~~
wyager
> What does the world look like in radar?

Depends on the radar system. Some are distance only without direction. Some
are 1D (a line, usually horizontal) and some are 2D. Many objects are
partially opaque, which is confusing. Resolution is very poor compared an
optical device of the same size.

> Could multiple radar emitters and receivers be used to create a phased array
> and improve resolution?

Yes. However, this is currently bulky and expensive (in dollars and in compute
power). Thankfully, it looks like capitalism is coming in to the rescue here
and miniaturizing the everloving shit out of complex radar arrays for human
interface tech. This should be usable for vehicles as well.

~~~
Animats
All the automotive radars are phased array devices, and have been since the
Eaton VORAD of the late 1990s. No moving parts. They're usually 1D scan
(horizontal) only, although 2-axis scanned automotive devices exist.

------
robojamison
> The car computer will then silently compare when it would have braked to the
> driver action and _upload that to the Tesla database...whether Autopilot is
> turned on or off_ , then that object is added to the geocoded whitelist.

(emphasis mine)

This has interesting privacy implications. I am not a Tesla owner, but I
imagine that by enabling Autopilot you consent to providing Tesla with
diagnostic, error, and sensor data. But what about those who have not enabled
this feature? Their Tesla will automatically phone home with data regarding
their location and surroundings regardless of whether or not they have
consented to this?

~~~
jeffwass
What surprises me is how (at least here in HN) there was a general feeling of
annoyance when Apple tracked user locations with Apple Maps to identify
traffic patterns, even though Google did the same for Google maps.

But now with Tesla user-tracking, people seem to be actively psyched at being
tracked by Tesla.

~~~
soheil
If Facebook or another company with goals that may not be clearly noble did it
people may question their motives more. I also believe these is a natural
higher level of trust in people like Elon Musk who have more praise worthy
missions set for their companies than let's say someone like Mark Zuckerberg,
ie. Elon wants humans to have a backup plant and is taking actions to save
this planet, he seems to have dedicated his life to that mission, Mark maybe
not so much.

~~~
CamperBob2
Musk also isn't on record as calling his customers "dumb fucks," the way
Zuckerberg is.

~~~
renaudg
What are you on the record saying when you were 19 ?

------
_ph_
It never stops to amaze how software improvements can greatly expand the
capability of a given hardware - all this is done on top of the 2014 autopilot
hardware. That is, what I like about software - there seem to be very few hard
limits you cannot work around with a clever new approach.

In the race to the self-driving car, Tesla now has one big advantage: they
have tens of thousands of cars with the autopilot hardware driving around
every day. This gives them a huge lead in the amount of data about their
software performance - just comparing what the radar sees and how the human
drives in any situation should make a difference.

~~~
jayjay71
As much as I love writing clever software to work around hardware limitations
(I believe it's what makes videogame programming of the 80's and 90's
fascinating and led to better creativity and better games), I once had the job
of writing software for "broken" hardware. By this I mean I was in charge of a
computer vision algorithm, and the camera was physically incapable of taking
the images necessary for the algorithm to function. It's worth mentioning this
was for a self-driving car.

Tesla will never come close to a stage 4 autonomous vehicle with the hardware
rigs they're currently selling. That said, it'll be interesting to see what
improvements they can make with software. Given their over-promise and under-
deliver history though (which arguably killed someone), I'll take their
marketing with a grain of salt.

~~~
_ph_
If the hardware does not work in the first place, then there is only so much a
software can do about it. But here we are talking about a system which works
and gets improved.

Tesla is very explicit about the limitations of their current system. So the
autopilot accident seems to be mostly about the owner not really understanding
what the current autopilot can do and what not - after all, in the car it is
just called "autosteer".

There is also already talk about the autopilot 2.0. This consists of augmented
hardware, e.g. 3 different front-facing cameras. Only with that hardware Tesla
is trying to reach level 3 or 4. The 8.0 update is about enhancing the quality
of the existing autopilot but not about reaching new levels of automatic
driving.

------
patcheudor
I'd be interested to hear how the radar handles other radar signals. Given the
use of police radar, radar detectors, and radar based collision avoidance like
what's found in the rear tail-lights of some Ford F150's and of course are
used on a fair number of Audi's it would seem the environment could get noisy
at times. Yes, the Tesla radar could operate at a specific frequency that
would minimize interference, but what happens when a bad actor decides to
intentionally "blind" that radar signal? I assume given this is a life
critical system that it would have countermeasures, perhaps utilizing a LIDAR
or camera based backup?

~~~
Matthias247
I don't know how this particular radar system works, but in general you can
modulate radar signals with a special encoding, so that only the sender can
receive and interpret it. For all others the signal is juse noise. At
university we worked with m-sequence based radar systems that have these
properties. This would minimize the possibility that someone (accidently or
not) can send signals that you misinterpret. Depending on the remaining
possibility you might still want to take same countermeasures.

~~~
shas3
Yes, but if you flood it with white noise, it'd screw up any transmit
waveforms. The only alternative is to have a software radio that is flexible
enough to switch bands when it encounters noise.

~~~
patcheudor
In terms of the SDR stuff, I'm thinking more about it being in the hands of a
threat actor. One of the things that Tesla nor Google have had to contend with
to date in their trials are intentional disruption and subversion of their
systems and I only hope they've hired security engineers to consider more than
just IT stack vulnerabilities like those explored by Charlie Miller & Chris
Valasek
([https://www.youtube.com/watch?v=OobLb1McxnI](https://www.youtube.com/watch?v=OobLb1McxnI)).
A great number of jobs are at risk from this technology from long haul truck
drivers to bus and cab drivers. This will certainly result in tensions which
could result in threat actors going after sensing systems. Readily available
radar frequency SDR in those conditions really could be used for harm.

------
msoad
The fleet of Tesla cars on road is an advantage that Uber and Lyft have over
Google. They can deploy cars with a lot of sensors on the road AND make money
off of it for the most part!

If data is the differentiating factor in this game, Google has less of it!
Which is interesting position for Google to be at!

~~~
objclxt
> If data is the differentiating factor in this game, Google has less of it!

If you're talking about Uber and Left, I think Google currently has them beat
in terms of data on a global scale, given how long they've been collecting
data for Maps. You're talking about a future scenario where Uber and Lyft have
rolled out significant numbers of sensor laden cars, but right now they don't
have that - and who's to say Google won't have another approach by then.

~~~
toomuchtodo
Google's map data is in no way comparable to the data collected by Tesla's
fleet (expert trained by vehicle owners), which by the way is gathering a
million miles of experience every 10 hours.

~~~
frankchn
True, but Google's Street View cars have 360 degree cameras and LIDAR systems
on them and I assume the data is saved at full fidelity and sent back to
Google.

Teslas definitely have many more miles on them, but I don't think the cars are
sending back every single frame captured by its cameras back to Tesla HQ.

~~~
Terretta
They only have to send the _interesting_ ones.

~~~
Piskvorrr
Which is a bit circular - lot of things are easy if you can detect "
_interesting_ ", but like with pornography, this is one of the "I'll know it
when I see it" AI-hard, human-easy tasks.

------
ArkyBeagle
It's quite strange to view how ... flexible the requirements-level changes
are. As someone steeped in safety and human life critical software
development, this seems very odd.

This is especially true of "Interface alerts are much more prominent,
including flashing white border on instrument panel." This has been a huge
thing in aviation automation for like ... forever.

~~~
Fej
"Autopilot" is "experimental" so whenever there's a problem, they blame it on
the driver.

I wonder if they have heard of Six Sigma.

~~~
Piskvorrr
There is currently no legal framework which would allow anything else than
"driver is responsible for vehicle". (I suppose that might change soon)

~~~
JumpCrisscross
More accurately, there is no true legal system _period_. Tesla's playing PR to
avoid a damaging precedent.

~~~
Piskvorrr
Surprisingly correct: USA did not sign the Vienna Convention on Road Traffic,
as the only western country. Special snowflake as usual :D

------
espadrine
I always assumed that humans evolved eyes adapted to the wavelengths of light
because it gave optimal information to avoid collisions. (Well, except for
glass panes; we're not genetically optimized for those.)

It feels scary to discard millions of years of evolution and go with radar-
first, but as always, time will tell.

~~~
_ph_
The biological eye also developed based on the availability of natural
electromagnetic radiation. And we are using most of the spectrum which easily
passes through our atmosphere. Radar waves are nice in that they pass through
rain and fog, but they are not well reflected by animals. So for surviving in
the wild, they are less useful, but on the streets where most cars are metal-
heavy they work much better. They do however not give a very well resolved
picture. The radar cannot really distinguish between a bridge and a high
trailer. The new software tries to work around that by using data from other
Teslas passing the same road section before.

~~~
Pyxl101
> The radar cannot really distinguish between a bridge and a high trailer.

Are there any visualizations available of what these radar patterns look like?

I wonder whether a human looking at the radar visualizations could reliably
discriminate between them. I have to suspect that we could, but I've never
seen such a visualization and don't have a good sense of what kind of detail
it includes.

~~~
semi-extrinsic
Here is a really great writeup from a guy building his own radar. Look at e.g.
the SAR images right at the end.

[http://hforsten.com/homemade-synthetic-aperture-
radar.html](http://hforsten.com/homemade-synthetic-aperture-radar.html)

------
henryw
"This is where fleet learning comes in handy. Initially, the vehicle fleet
will take no action except to note the position of road signs, bridges and
other stationary objects, mapping the world according to radar."

Wow, glad to see that they are using big data and machine learning. If all
those 400k orders go through, there will be a network effect in favor of
Tesla.

~~~
Smaug123
Perhaps Tesla might release the data. Musk is all about improving humanity;
selling cars is a means to an end, rather than the end itself, and that end
could be hastened if every manufacturer could access (and augment?) the
database.

~~~
bobsil1
Musk is all about saving humanity, he's apocalyptic. Probably rightly so.
Improving also, but not what really drives him.

------
pcl
Two really interesting notes buried in the release notes at the end:

> With further data gathering, car will activate Autosteer to avoid collision
> when probability ~100%

> Curve speed adaptation now uses fleet-learned roadway curvature

~~~
Theodores
...imagine this in racing. Saw some races over the weekend and the racing line
varied considerably between cars and drivers, not like F1 where every driver
is that good. I imagine the wider population have an even wider idea of what
the 'racing line' is. In theory the Tesla car could take 'better than Senna'
lines through ever curve, avoiding the crashes and also optimising efficient
regenerative braking. I look forward to this and I am glad the Tesla brain is
learning from 'the fleet'.

~~~
scott_karana
BMW was doing that circa 2011: even brought it to the Top Gear track.

So yeah, there's a precedent :)

------
Fej
If I were to ever buy a Tesla, could I turn off data collection?

If not... that would turn off a lot of privacy-conscious people, which Tesla
doesn't tend to attract at its current prices but may become relevant as they
come out with cheaper cars.

~~~
JulianMorrison
If you have a mobile phone, you are already broadcasting your location. Go to
google maps, open the sidebar menu, select "your timeline".

Sorry. Privacy kind of went byebye.

~~~
serf
I don't understand arguments like that.

"You already have heart disease, who cares about colorectal cancer? Have a
cheeseburger!"

Privacy isn't some binary choice, and having _less_ data collected means
_less_ data being spilled when X company has a server compromised.

------
serf
Every time I am reminded about how Tesla can update their cars in the field I
always imagine the stress in the responsibility of securing a network like
that, and the risk a compromise carries. Heavy work for some team.

------
sundvor
The "simple" radar cruise control in my 2014 Mazda 3 Astina (ex demo; MT) is
an amazing experience. I'm finding it remarkably accurate, even picking up
objects (motorcycles, bicycles) that they explicitly state will not be
correctly registered. The simple, yet effective HUD indicates current
following distance (as a side note, most drivers are way closer than 2s).
Coupled with the visual AEB system, I have autonomous braking for traffic and
emergencies. I still have to steer, however the lane departure system warns if
I'm exiting my lane without indicating at +65km/h.

It might not be anything like Tesla autopilot, but it's still a pretty sweet
taste of the future. So stands to reason more can be done with it; I wish I
had the funds for a Tesla... Maybe one year :)

------
rajathagasthya
Slightly off topic, but are the software updates optional or mandatory in
Tesla cars? As I do with phones, I hold off on updates until I know it's
stable and has no major bugs. Serious bugs in a car software could be fatal.

~~~
ArkyBeagle
I can't even begin to imagine how that would be done in this case. While the
downside of Bad Things in a Tesla car is not nearly as spectacular as in a
late-1950s experimental aircraft, it seems all owners are in effect test
pilots.

What I'd like to see is an emphasis on stupefyingly comprehensive test
vectors.

------
mrfusion
Does the car still use optical cameras? I'm not understanding how those factor
in.

~~~
zwily
It has to for lane keeping. Radar can't see painted lines.

------
gambiting
Serious question - if there are two Teslas, travelling in opposite directions,
will their signals not interfere with each other?

~~~
oasisbob
One of the easiest ways would be to use a randomized pulse repetition
frequency? The other Tesla would be recognized as a discrete interference
source.

Seems this is a well-known problem in airborne radar:
[http://www.dtic.mil/dtic/tr/fulltext/u2/a402557.pdf](http://www.dtic.mil/dtic/tr/fulltext/u2/a402557.pdf)

------
jacobevelyn
> _Something made of wood or painted plastic, though opaque to a person, is
> almost as transparent as glass to radar._

combined with

> _...we now believe [radar] can be used as a primary control sensor without
> requiring the camera to confirm visual image recognition._

seems like they're now intentionally ignoring the possibility of wooden and
painted plastic obstacles?

~~~
XorNot
Sounds the opposite to me: if the radar says "stop" then the car will do it.
It won't go though if the camera says stop.

------
etendue
> The radar was added to all Tesla vehicles in October 2014 as part of the
> Autopilot hardware suite, but was only meant to be a supplementary sensor to
> the primary camera and image processing system.

I guess I'm surprised that what sounds like a large change in ConOps can be
rolled out as an upgrade across a fleet in such a short period of time. It'd
be fascinating to hear what sort of V&V had to be done, and how it was
accomplished so quickly, to make this happen.

------
amq
I hope they take their time and not rush the update. This is probably the
biggest change since introducing Autopilot. Also, Tesla is now in a very
dangerous moment where a serious failure can bury Autopilot.

------
JulianMorrison
Interesting that they are silently crowdsourcing a map overlay. When you drive
a route in your Tesla, you're playing the part of a Google mapping car for
them.

------
mrfusion
So are they saying radar wouldn't see a tree fallen across the road? Is there
a backup system for that kind of thing?

~~~
maxerickson
The human. At least, given the technology that Tesla is deploying here, the
driver would be expected to continue to pay attention to the road and to brake
for obstacles.

I mean, a really cynical view looks at the part where it works better as a
smokescreen for the part where it much more aggressively monitors driver
attention.

------
mrfusion
Can someone explain the other release notes? Most of them seemed terse and I
didn't understand what they were saying?

------
mrfusion
I'm not understanding how they surmount the soda can problem?

~~~
baronvonsp
A single radar return at just he right angle might make the soda can look
huge. But with multiple sensors moving relative to the soda can and sampling
at 10Hz, you end up able to form a "real" composite of the object

------
kmonsen
Wonder when next HW is coming, supposed to be this year.

~~~
LAMike
My guess is the same time that the Model 3 launches (July/August)

~~~
scott_karana
Maybe, but considering the scrutiny on Tesla right now, I think they'd prefer
to launch any new hardware ASAP, irrespective of the marketing benefits of
launching it with the 3.

------
mrfusion
makes me wonder what's going on with radar detectors these days? Are they just
100% false alarms with all the cars using radar?

------
foobarqux
It is frightening how little the press release addresses the many catastrophic
failure modes.

~~~
danhak
Just needs to have fewer failure modes than humans.

~~~
foobarqux
We have no evidence that the current Tesla system causes fewer injuries or
fatalities. Moreover, the alternative isn't non-automation but other
reasonable automated designs, ones that do address the failure modes.

------
maxcan
> The net effect of this, combined with the fact that radar sees through most
> visual obscuration, is that the car should almost always hit the brakes
> correctly even if a UFO were to land on the freeway in zero visibility
> conditions.

Unless he means "UFO" in the formal, aviation sense of an unidentified radar
contact, I think Mr. Musk is severely underestimating alien stealth
technology. I can't quite wrap my head around the notion that an advanced
species could perfect interstellar travel but somehow be incapable of stealth
technology.

~~~
ansgri
Maybe they simply never needed it, their primary sensor being gravitational
wave GrDAR?

