
The Camera Is the Lidar - nthuser
https://medium.com/@angus.pacala/the-camera-is-in-the-lidar-6fcf77e7dfa6
======
Animats
The hardware is described, vaguely, at [1]. It's a rotating drum scanner with
16 or 64 lasers. $12,000 for the 64-laser model. $24,000 for the upcoming
model with 200m range. Still a long way from a useful auto part.

There are about a dozen companies in this space now. Nobody has the price down
yet. Continental, the European auto parts company, is the most likely winner.
Quanergy made a lot of noise but didn't ship much.[2] There's a conference on
automotive LIDAR this month in Detroit.[3] Many of the exhibitors are major
semiconductor packaging companies, with various approaches to putting lots of
little LIDAR units in a convenient package at a reasonable price.

[1] [https://www.ouster.io/faq/](https://www.ouster.io/faq/) [2]
[https://news.ycombinator.com/item?id=17755183](https://news.ycombinator.com/item?id=17755183)
[3] [http://www.automotivelidar.com/](http://www.automotivelidar.com/)

~~~
deepnotderp
Why Continental?

Flash lidar seems to be a fundamentally broken concept to me wrt range.

~~~
Animats
One might think that, except that Advanced Scientific Concepts, which
Continental bought, has had it working for a decade.[1] Their units work fine,
but are expensive. They're mostly sold to DoD and used for space applications.
The Space-X Dragon spacecraft uses one for docking.

There's a tradeoff between field of view and range. Automotive systems will
probably include a long-range narrow field of view unit and a shorter range
wide field of view unit.

Flash LIDAR has some advantages. No moving parts. Can be fabbed by
semiconductor processes. The one big laser is separate from the sensor array,
which helps with cooling. Also, you can spread the outgoing beam, which helps
with eye safety. (Eye safety involves how much energy is in an eye iris sized,
1/4" or so, cross section of the beam. If the beam is spread out, energy
density is lower.)

[1]
[https://ieeexplore.ieee.org/document/7268968/](https://ieeexplore.ieee.org/document/7268968/)

~~~
toomuchtodo
What do you see as the key challenges in driving down the cost of flash LIDAR?

~~~
dllu
The light from the flash spreads out with the inverse square law, so you get
much less signal compared with collimated laser beams. To compensate for that,
you need much higher power. To get more power without blinding people, you
need to use 1550 nm. This requires large arrays of exotic gallium arsenide
semiconductors, which are expensive.

------
flyinglizard
Why is this better than separate LIDAR and camera?

Because you're collecting NIR ambient light, your optics are wideband. Meaning
that daylight would have a more pronounced negative effect on system range
(easier to saturate the photocells). It's also low resolution (as most LIDARs
are), and there is no color segmentation data.

In an automotive application, I can't see a justification to unify both visual
and LIDAR into a single sensor, rather than having an extrinsically calibrated
array of sensors. You can improve the calibration out of the data over time if
you're very concerned about system stability.

It seems like a nice party trick, but the vehicle LIDAR game focuses on solid
state long range units, as this will be what gets into mass production. The
visual band imagers in the car are a given for many other reasons anyway.

~~~
deepnotderp
It's better because there's no need for calibration, you always have perfect
calibration.

Solid state lidar has issues. The cofounder of Ouster, Angus Pacala previously
cofounded Quanergy, a solid state lidar startup.

~~~
flyinglizard
Solid state LIDAR certainly has issues - but someone is going to solve those
and this is what will get into automotive, definitely not $10k units with
moving parts.

There was an announcement on a cooperation between BMW and Innoviz (an Israeli
maker of solid state LIDARs) with Magna being their OEM sponsor.

I'm not sure calibration is that big of a deal for this application. Sensors
are going to be calibrated and tested in the factory or at a module level
regardless, and the accuracy requirements in automotive are much lower than
consumer products using similar technology.

You can't overcome not having colors (traffic lights, anyone?), limited
ranging distance or sensor saturation due to ambient conditions.

------
barbegal
I'm sure this works well in bright light but I'm sceptical that this can
perform at all well on overcast days or at night. The OS-1 device spins at
10Hz and the LIDAR samples 2048 points over one 360 degree revolution. This
means each column of pixels is sampled at 1/20480 of a second. To sample that
fast requires a lot of light which is fine on a sunny day but on a cloudy day
you can have 100 time less light. And at night you would have no ambient near
infrared light at all.

~~~
post_break
Could you have a camera without the IR filter and then use IR leds to make up
the difference? You would have to worry about other vehicles doing to same and
blinding the cameras somehow but that might be a solution.

------
amelius
The problem is that deep learning should not be allowed in safety critical
systems, because (1) the accuracy is always less than 100% even in known test
situations, and (2) we don't know how it works and under what conditions it
breaks down.

~~~
simonsarris
I was in a car crash two years ago where a man went into a diabetic
fit/seizure and sped through an intersection, ultimately hitting a building
and my car and killing himself in the process. It is too bad his car did not
have some of this deep learning that is not 100% accurate.

We don't know how humans work and under what conditions they break down,
either.

~~~
nradov
There is no evidence that deep learning would give better performance than
other collision avoidance algorithms in such a scenario.

~~~
lucb1e
But that doesn't mean we shouldn't try. I was agreeing with GP (u/amelius)
because I had the same idea when reading the post, but the parent of your
comment (u/simonsarris) makes a good point: we might not know deep learning as
well as we might like to know it, given that it is being used in applications
that have the potential to kill, but we also don't know our own brains that
well.

Even if we don't understand deep learning to the degree that we would like, we
can observe its safety record and compare it to humans'.

------
kumarvvr
A semi-off topic question. Is it not possible to get an accurate depth map
based on a two camera stereoscopic setup? Like human eyes? Perhaps combine it
with video processing to isolate objects at different depths.

~~~
kyrra
I was watching a talk from Cruise that mentions this. The main problem with
cameras is dynamic range. Dealing with different lighting conditions that can
change quickly is hard (the sun is really good at washing out colors). Lidar
doesn't care about the current lighting conditions.

[https://youtu.be/s-8cYj_eh8E?t=22m39s](https://youtu.be/s-8cYj_eh8E?t=22m39s)

~~~
sjwright
Also heavy rain would be a problem for regular cameras. Not just seeing
through the airborne droplets, but also (at a guess far more significantly)
the water directly in contact with the windscreen causing severe random
distortions.

~~~
sbr464
I built a hacky prototype, combining:

\- FLIR thermal camera

\- 3 different small cameras manually set at different settings, models chosen
for their qualities handling light levels.

Those 4 live feeds were fed into a small black magic design quad layout
device, that turned them into a single hd feed via hardware/real-time. That
was fed into a hardware capture, that stacked the quad arrangement, applied
some other filters and did hardware compression. At that point almost no
latency was introduced but had a nice working base video feed. That was fed
into the Linux box for processing.

The quad device created a sort of super hdr video, and the thermal layer took
it to the next level. All of the cameras had drawbacks, but combined they were
minimized.

------
zerealshadowban
This is so brilliant, I'm considering completely switching the type of
software/hardware work I've been doing for the last decade.

------
sytelus
The blog post is very badly written to figure out what really these folks are
doing and what sets them apart. Here’s my guess: they are detecting laser
bounces using modified camera and using that camera to generate visual image
at the same time as point cloud. It’s still a mechanical moving lidar at 10hz,
range is 120m and resolution of 2048 horizontal beams per 360-degree and 16
beams verticle looks pretty good, although not completely out of the league.

------
bsaul
Something i don't understand : i've seen those kind of videos (i think it was
nvidia or maybe waymo) where a signal (either camera or lidar or both) is
processed in real-time and displays boxes around cars, street lights, etc. It
seemed to be that detouring real-life objects from sensors in real-time has
been working for a long time now.

What does this bring that's new ?

~~~
dllu
Estimating depth and 3D bounding boxes using cameras only, while possible, is
much less accurate than using actual range data from a lidar.

On the other hand if you fuse lidar and camera data such as with Waymo and
others, there may be issues with the sensors being out of sync (as they run at
different framerates, and the lidar continually spins) or physically offset
(leading to parallax issues). Dealing with such issues is very difficult.
Having a single sensor output both accurate range information and camera data
makes it much nicer to work with.

------
dllu
Relevant discussion with some comments from Angus on reddit:
[https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the...](https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the_camera_is_the_lidar/)

------
haney
So I'm not up on the Lidar industry but $12k for a sensor seems really
expensive, but then again from my casual observations Lidar is really
expensive. Is there a physical / first principles reason this is true or is it
just really new technology?

~~~
patrickg_zill
The most general answer to your question is Peter Drucker's (famous management
guru) observation that every doubling of production of an item (over the
lifetime, not yearly) resulted in a cost reduction of 20-30%.

So right now, with very few LIDARs produced, we have a high price, which will
start dropping as more are produced.

You might find this interesting: a single transistor used to sell for roughly
the equivalent of $8 USD in today's money; today the cheapest ones are 6
_Cents_ USD (price checked today from Mouser.com) in qty 1 pricing...

[https://spectrum.ieee.org/tech-
talk/semiconductors/devices/h...](https://spectrum.ieee.org/tech-
talk/semiconductors/devices/how-much-did-early-transistors-cost)

~~~
StavrosK
At quantity 1, though, most of the cost is from the person who has to package
it. Qty 100 will give you a much more accurate price.

~~~
TheSpiceIsLife
You can buy 9.6 billion transistors for AU$599 (US$430) in the form of an AMD
Threadripper 12 Core 1920X.

That's AU$0.000,000,062,395,833 (US$0.000,000,044,791,667) per transistor;

------
barrystaes
I get really excited about this technology, yet the not-made-here-syndrome
force is strong in this one. I wonder if theres an EU equivalent?

~~~
dllu
There are many EU lidar companies. SICK, Ibeo, Pepperl+Fuchs, Osram, Innoluce,
Blickfeld, and so on.

However, none of those matches the capabilities of the Ouster OS-1 exactly.

------
akeck
Could hundreds of self driving cars cruising around a city with LIDAR affect
the eyes of pedestrians?

~~~
dllu
It would not affect the eyes of pedestrians.

The Ouster OS-1 in the article, as well as all other automotive lidars that I
know of, are class 1 laser eye-safe, meaning that it is safe even if you put
your eye right up to it for hours.

The power also decreases dramatically once you get far away from it, since the
laser beams spend most of their time pointed in different directions, and the
collimation is not perfect.

------
thanatropism
Is it just me, or are we seeing yet another impressive leap in Computer Vision
that's soon going to be hyped as an incremental step into Skynet?

~~~
21
Unless you think Skynet is impossible, all impressive leaps are an incremental
step towards Skynet.

~~~
wpietri
Not necessarily. My expectation is that Skynet is highly unlikely, a side
branch we probably won't take.

Think of the 1920s-1950s version of robots, for example. They were machines
shaped like people and that acted like people. In retrospect, they seem not
scary but silly. The human shape isn't particularly useful or easy to build;
our most common robots are vacuums shaped like hockey pucks.

Skynet is another "what if machines acted like people" fairy tale. It makes
sense if you imagine yourself as a computer that wakes up; we wake up all the
time, so it seems normal to us. But self-awareness and self-preservation are
biological systems that evolved over very long time scales. Those are
intricate systems, again not really useful or easy to build. And also not
likely to randomly occur.

It could be that we'll build those kinds of systems, of course. But I think it
will take a long time to get them right, and then it's not really the skynet
story, it's the mad scientist with the robot army story.

