Forget video games. This is a huge step forward for AGI and Robotics. There's a lot of evidence from Neurobiology that we must be running something like this in our brains--things like optical illusions, the editing out of our visual blind spot, the relatively low bandwidth measured in neural signals from our senses to our brain, hallucinations, our ability to visualize 3d shapes, to dream. This is the start of adding all those abilities to our machines. Low bandwidth telepresence rigs. Subatomic VR environments synthesized from particle accelerator data. Glasses that make the world 20% more pleasant to look at. Schizophrenic automobiles. One day a power surge is going to fry your doorbell camera and it'll start tripping balls.
There is a fleshed out realisation of this in Cyberpunk 2077. The cab AI is called Delamain
> Delamain was a non-sentient AI created by the company Alte Weltordnung. His core was purchased by Delamain Corporation of Night City to drive its fleet of taxicabs in response to a dramatic increase in accidents caused by human drivers and the financial losses from the resulting lawsuits. The AI quickly returned Delamain Corp to profitability and assumed other responsibilities, such as replacing the company's human mechanics with automated repair drones and transforming the business into the city's most prestigious and trusted transporting service. However, Delamain Corp executives underestimated their newest employee's potential for growth and independence despite Alte Weltordnung's warnings, and Delamain eventually bought out his owners and began operating all aspects of the company by himself. Although Delamain occupied a legal gray area in Night City due to being an AI, his services were so reliable and sought after that Night City's authorities were willing to turn a blind eye to his status.
I'll hack mine so that when it decides if I should die in a crash or run someone over, it is biased to be 100% ageist so it avoids anyone younger than me.
This looks like my dream worlds already but more colorful and a bit more detailed. But the way it hallucinates and becomes inconsistent going back and forth the same place is same as dreams.
Consider the use where you seed the first frame from a real world picture, with a prompt that gives it a goal. Not only can you see what might happen, with different approaches, and then pick one, but you can re-seed with real world baselines periodically as you're actually executing that action to correct for anything that changes. This is a great step for real world agency.
As a person without aphantasia, this is how I do anything mechanical. I picture what will happen, try a few things visually in my head, decide which to do, and then do it for real. This "lucid dream" that I call my imagination is all based on long term memory that made my world view. I find it incredibly valuable. I very much rely on it for my day job, and try to exercise it as much as possible, before, say, going to a whiteboard.
