In my mental model you don't need a loss function for this. It's just, given these potential short term goals predict the future state of the game 1 minute from now. Then choose the short term goal that maximizes uncertainty for the future. The idea being that you have a game playing bot that learns to do "things" in the game world, and a game state encoding bot that learns to describe the game world. I'm not suggesting you train the bot TO do the least certain idea, I'm saying you train the bot BY doing the least certain idea; and update your understanding of the world to be more certain about that choice in the future.
The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path.
The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.
Your last paragraph reminds me of Hindsight Experience Replay[1] where they use goal-conditioned policies that learn how to reach essentially any state. The idea is that, even if you don't reach your true goal, you've learned how to reach some other goal, so you can successively learn to navigate the environment until the end goal is easy. It's a really nice paper, highly recommended!
The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path.
The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.