
Differentiable Monte Carlo Ray Tracing Through Edge Sampling - jonbaer
https://people.csail.mit.edu/tzumao/diffrt/
======
317070
I've done something similar in beginning of 2017:
[https://openreview.net/pdf?id=SyEiHNKxx](https://openreview.net/pdf?id=SyEiHNKxx)

I had the whole physics engine + renderer differentiable, in Theano. I used it
to do control problems though (so using the image to infer which actions to
take), not for system identification (i.e. given the image, what are the
parameters of the scene). But yeah, I could backpropagate through the renderer
too:
[https://twitter.com/317070/status/821062814798331905](https://twitter.com/317070/status/821062814798331905)

I think more research should go in this direction, backpropagating these
physical systems will empower a lot of powerful optimization methods. There is
no reason why a physics engine should be treated as a black box. We know the
equations and they are differentiable. I'm glad there will now be dedicated
code for the gradient of a full blown renderer :-)

~~~
modeless
I saw this work when you published it. It looks really impressive! Have you
continued to work on it? Theano is no longer being developed, so I guess you'd
have to switch to something else. What are the next directions to take for
this approach?

~~~
317070
I tried to submit it to two conferences, twice rejected. Had to leave academia
after funding ran out (did get my Phd). Moved the manuscript to a journal and
just last week got a really bad review (saying this had been done in the
nineties already) and it will probably be rejected again. All of this is not
encouraging my old research group to continue in this direction.

So, the paper and code will likely end up in the garbage bin of academia. And
Theano quitting was also not exactly helping indeed.

Next directions are exactly what these people do with their ray tracing. The
equations of all of these things are rather simple and almost always
differentiable. Someone just needs to code them up and start experimenting
with what is achievable when you have these exact gradients.

If someone is looking for a technical area for a Phd/startup: such a physics
engine with a focus on optimization (so both system identification, system
control and system optimization with gradient descent, making all internals of
your engine differentiable) is worth millions.

~~~
modeless
Wow, I am sorry to hear that. I am more interested in differentiable physics
than differentiable rendering; it seems promising for training controllers for
animated game characters or even real-world robots with transfer learning. I
might try my hand at it but I think there is a whole literature on rigid body
dynamics that I would have to read up on first.

I would be interested to hear if you have ideas for specific application areas
that would be feasible to tackle as a startup. Animated game characters is the
best I've got but I'm not sure it's worth millions.

------
Geee
There are more example on the supplementary page:
[https://people.csail.mit.edu/tzumao/diffrt/supplementary_web...](https://people.csail.mit.edu/tzumao/diffrt/supplementary_webpage/)

------
gh02t
This is interesting, there has been research work on doing something similar
in Monte Carlo particle transport, which is the same idea as ray tracing but
with different physics (higher energies, different types of particles, more
complex physical interactions like incoherent scatter etc). Still reading, but
this article appears to be based on the same general idea albeit with a
different objective.

Differentiation of particle transport Monte Carlo codes is focused on
different types of tallies and subsequently computing their sensitivities. The
inverse rendering problem in 5.3 is actually also broadly applicable to fields
like nuclear security and emergency response.

------
skratlo
ELI5 please

~~~
saagarjha
I think it's doing this: [https://en.wikipedia.org/wiki/Gradient-
domain_image_processi...](https://en.wikipedia.org/wiki/Gradient-
domain_image_processing)?

~~~
remcob
Don't think so. This solves problems in the 2D domain, but the paper solves
problems in the 3D domain.

------
nmca
This is PyTorch compatible, which is dope.

