
Ask HN: Autonomous Cars Simulation - coderunner
There&#x27;s a couple companies in the business of using a 3d simulation for training autonomous cars like Waymo, Cruise, Nvidia, and Applied Intuition. I don&#x27;t quite understand their product though.<p>1. Are the trained object detectors in the simulation applied to real world data also or is only the part that makes decisions transferred to the real vehicle (e.g. it&#x27;s safe to turn left here) while detectors trained on real world images of cars, people, etc. used?<p>2. Tangentially, I thought that in general detectors trained on computer generated images was not very applicable to real world images. eg training on a bunch of images of 3d modeled humans won&#x27;t work well with testing on pictures of real humans. Is this not true?
======
hacoo
I work on full-3D simulations (using a game engine) for an autonomous car
company. I can't speak for every AV company, but in my experience, simulators
are used far more for testing than 'training'. The appeal of using a 3D game
engine for simulation is that you can create inputs to the car's perception
system. Without this ability, you're stuck either replaying recorded data, or
spoofing out perception and only testing planning/controls and down. These two
approaches are actually extremely powerful, so the vast majority of AV
simulation testing is not done in full 3D.

There are some situations where 3D simulation is useful, though. First, it
allows you to run your AV software in its entirety (i.e., not spoofing
perception), making for a very complete integration test. A 3D sim can capture
complex, interesting occlusions that other sims cannot. Another fairly common
use case is experimenting with new sensor setups before they're added to the
car.

As for training, it's mostly research at this point. I think there's promise
in using synthetic data to supplement real-world data training data for
perception systems.

There are a number of companies trying to market simulation 'platforms' to AV
makers. I think there's the potential for one of these products to gain
traction -- but it's a difficult sell. AVs are enormously complicated, a 3rd
party product would need to both beat in-house sims and support a lot of very
specific (and likely propriety) AV features.

~~~
coderunner
Thanks for answering! Does that mean the perception system that's trained on
real world data does well on the synthetic data coming in during testing? Or
are the perception systems' training data kept separate (synthetic data for
training the perception system used when testing in the simulation and real
world data for training the real world perception system)?

And yeah, I've come across quite a few companies working on simulation
platforms in my reading (cognata, metamoto, applied intuition, etc.) which
made me interested in what they're selling but actual details are quite sparse
on their websites.

------
Datenstrom
Simulation to real training is an active area of research and I'm not aware of
it being used anywhere in production for critical systems. I have not seen
anything better than the "Learning Dexterity"[1] paper that was published last
year.

[1]: [https://openai.com/blog/learning-
dexterity/](https://openai.com/blog/learning-dexterity/)

~~~
coderunner
That was my impression as well. Maybe the idea is get the models trained in a
simulation to a satisfactory level and work on transfer of the model to real
world data in parallel? There's an awful lot of money going into the first
part if there's no good answer to the second part right now though.

------
natch
Taking just the images subset of this, but it should apply for other types of
data as well:

It doesn’t have to be computer-generated images. It can also be
computer—altered images (think n° rotation, blurring, cropping, etc.) which
should work pretty well in part because real world images are sometimes
rotated, blurred, cropped, etc.

~~~
coderunner
I just found this paper related to that actually

[https://arxiv.org/abs/1901.07849](https://arxiv.org/abs/1901.07849)

From what I can find out seems like all the companies are using unreal or
unity for rendering the graphics though.

