One thing I haven't seen mentioned in previous discussions of this and NeRF: how does the scene lighting effect new dynamic objects that are added to a scene?
For example, suppose this technique is used to render the static parts of a level in a video game (possibly even with some baked animation). What will happen when you put all the game characters, particles, bullets, and other moving things in?
Will we need a separate light layer for these? Or can the dynamic lighting be inferred?
Specular higlights are view dependent and the specular higlights seem to react properly when tumbling the virtual camera. It seems reasonable that it would be possible to do a reverse calculation to determine the location and intensity of light sources in a scanned scene. Once those are determined, they can be extracted from the scene and made global light objects in a scene in a DCC app or game engine. More than likely it would just be baked as one big IBL dome light.
These methods learn directly the radiance, in a way which is not lighting dependent. Therefore they would render the same and shadows are not taken into account. Current methods aren’t able to do this, but maybe some day we might discover methods which could infer and take into account some lighting parameters.
- "Gaussian splatting is pretty cool", 94 comments, https://news.ycombinator.com/item?id=37415478
- "Show HN: Real-Time 3D Gaussian Splatting in WebGL", 59 comments, https://news.ycombinator.com/item?id=37470611