When I think of turning sequences of images into gaussians, I think of the difficulty of getting generalizable information that can be re-rendered out of the pipeline; textures and lighting, basically. From the description at the top of the paper, where they mention adding dimensions for things like albedo, I got excited.
But the demos don't do any re-rendering / change of lighting / etc, so I can't tell if this paper is just a 'super high render quality at same training time' paper, which is of course great to have, or if it has a shot at being extended to get us scenes that can be adjusted as to lighting and texture in-engine.
This problem specifically (3D reconstruction with representation fitting) is really an overfitting nightmare, they just adapted to it not really overcame it. Nonetheless interesting work.
I get the feeling that some abstract geometric/topological approaches will let us get under fitted general pretrained NERFs that rapidly specialize to a given context, let you combine separate contexts into a composite scene, and maybe let's you give behaviors to entities in the scene.
At that point you're kind of trying to generically train over all possible scene graphs. But maybe some kind of geometric delocalization scheme lets you train on small fragments of larger scenes, transfer learning between scenes, and quickly represent and render generic scenes.
The point of a Nerf is effectively overfitting a neural network to a scene anyways (I think they say this in the original NeRF paper - or somewhere similar).
A Gaussian process fits a single high dimensional Gaussian, for example, by treating n observations along a single dimension as a n dimensional space.
Gaussian mixture models fit a large number of low dimensional Gaussians for example you might imagine 2D data generated by several 2D Gaussian superimposed.
This approach is just an example of the latter. It uses higher dimensional Gaussians to capture extra information from a scene, but not in the emulation of an infinite dimensional space in the way that defines Gaussian processes.
To add to a sibling comment, if you're interested in learning a bit about the both the Gaussian (as in a density estimator like Gassian Mixture Models, aka GMMs) vs Gaussian Processes (GP), I have some write-ups here: [1] and [2].
If you understand what a Gaussian Process is, you understand what a Gaussian Distribution is. This work estimates the parameters of many Gaussian Distribution in order to fit the approximate geometry of a GD to a geometry in a scene.
But the demos don't do any re-rendering / change of lighting / etc, so I can't tell if this paper is just a 'super high render quality at same training time' paper, which is of course great to have, or if it has a shot at being extended to get us scenes that can be adjusted as to lighting and texture in-engine.
Any experts care to chime in?