It's like its simulation model is confined to the renderable area. At the cost of increased computation, it would not be hard to generalize the technique to rendering a sphere centered around the POV to keep some of those artifacts at bay; however, the fundamental limitation of "object permanence" remains if objects can evolve chaotically as soon as they are blocked from view by another object. For that to be made stable, you really need a whole 3D model of the scene, and the cost increase there might be huge.
> While it can maintain coherent environments over short sessions, visual details and object permanence can still shift or drift over time—especially as a simulation runs longer, after multiple complex prompt edits, or when navigating throughout the scene. Complex physical interactions, real-time audio accuracy, rendering legible text, spatial accuracy, and long-term consistency are active areas of ongoing research.
If it was actually not hard, they would have made their demo with it. I don't know the technical details, but I expect a product reveal like this to show the thing at its absolute best.
People seem to be leaping to the conclusion that this is "product" as opposed to, fundamentally, research work, posted on a lab blog post, showing off a capability they expect to be increasing interesting over time.
Just because they happen to be throwing in the option to sign up to to play with the Genie 4 into an Ultra plan, doesn't mean that this is a material part of the value prop for Gemini.
Presumably, they can learn something from how niche Ultra users interested in tech demos engage with it. If that's an upside, it's really unclear it why it would make sense for "a reveal" to promise something too expensive to deliver to such Ultra users. (None of which means that the suggestion that you are responding to is technically reasonable or possible--just that it is unclear that one should dismiss the idea because one views this through the lens of a consumer "product reveal"
Just because OpenAI had to crawl back from Sora to focus resources, and Anthropic proudly proclaims in their filing that they have been avoiding promising creative opportunities for the same reason, doesn't mean that Google can't continue to do lab/research stuff.
didn't they show off an interactive demo with genie3 where you could paint on a blackboard and go out through the window and come back and the same was still written on the blackboard?
I am not sure you need something so complex. If objects on scene could be tagged with their location, and the camera movement tracked to identify when it brings the same objects back into scene, then the model can use its positional information to validate if the next rendered scene contains all the objects that should be expected.
Though, I am not sure well this retains temporal stability. I am just thinking about how in 3D rendering, objects are tracked as individual things and only culled, but all the data about them still exists.
Maybe the models need to reason as a 3D space first, then turn chosen objects into renderings?
> For that to be made stable, you really need a whole 3D model of the scene, and the cost increase there might be huge.
I suspect it is, because the 2D -> 3D models I've seen are terrible, but more importantly the 2D -> 3d models are terrible. Noteworthy because I suspect the number of people who have 3D scans laying around is tiny.
What you're describing is vaguely how video games work, to my understanding. There may be interesting parallels if that becomes a technique (like how distant objects are sometimes still rendered, but at a lower resolution/polygon count).
> What you're describing is vaguely how video games work, to my understanding.
Your understanding is broadly correct. There is a 3D world, and the GPU computes how it looks like from a particular viewpoint.
In essence, objects are painted back to front, so that in the final image every pixel shows the nearest visible object. The details get complicated, but the idea is simple.
> fundamental limitation of "object permanence" remains if objects can evolve chaotically as soon as they are blocked from view by another object.
they got this working decently well for video over a year ago, maybe closer to two years ago at this point, time goes fast. so i assume it'll translate to progress in Genie too
I think it's exactly this. The rendering capabilities in a true world model would include the entire domain it exists in. This takes a ton of processing power.
> While it can maintain coherent environments over short sessions, visual details and object permanence can still shift or drift over time—especially as a simulation runs longer, after multiple complex prompt edits, or when navigating throughout the scene. Complex physical interactions, real-time audio accuracy, rendering legible text, spatial accuracy, and long-term consistency are active areas of ongoing research.