This is intriguing. What would you need to model as a the continuous input to try and get from the timelapse? With the imagery, a model was made of the 3D scene in which was then used for the 2D final output.
I'm having difficulty imagining what this more in-depth model would represent, and how you'd either strategically take the clips to "paint" this model.
