Edit: The slides are fine but the gem is the YouTube lectures playlist!
In most modern games the human will invariably "win" eventually, so the "try to get most points before dying" approach makes less sense in general. But, _Speed running_ has become a very popular activity even in games you may not think of as being about "speed". I will be interested to see multi-purpose AIs take on speed running for two reasons:
1. The evaluation function is easier for the machine AND the human assessing it to understand, especially for "Any%" runs in most games, where the rule is basically you don't care how the game is supposed to be played, you just want to go from the start to the end in the least time using the controls (e.g. Mario Kart speed runs are often mostly about convincing the game that your shorter route counts as a "lap" of the track).
2. TAS (Tool Assisted Speed-running) is already a thing, TAS players don't compete with live human players, they choose and play back a precise sequence of inputs to cause the game to play out in a particular way, so they can do hundreds of frame-precise movements, or even influence hidden internal variables (with the effect that seemingly "random" behaviour from a game often proves to be deterministic). An AI can do everything the TAS player can, moment by moment, but it reacts to changes, which a TAS run cannot, on the other hand the TAS players often have deep insight into the implementation details of the game which would not be available to a general purpose AI player.
The one thing I would say about that is that speedrunning is a really, really difficult thing to do with a gradient descent approach; you can get really optimized with a particular strat, but getting substantially faster often requires drastically different strategies - which can only be discovered with a very broad knowledge of the game's systems, in areas which you will likely never see if your only experience with the game is speedrunning one particular strat exclusively. And exploration of the game's space during a speedrun would kill your time.
That's kind of what makes it interesting, no? You could argue that speedrunning requires not just observing a change in score from a change in inputs, but understanding, symbolically manipulating, and exploiting game mechanics.
Actually, just the idea of having an AI discovering glitches at all is an impressive use of AI.
Or perhaps even more difficult are the games that even humans struggle to play through to the end due to "cryptic gameplay elements" (of which there are a lot of examples in Atari games).
In Advent for example there are clear sub-goals and an AI can be programmed to try to achieve these, you actually get points in Advent (all versions I think?) for partial completion, so it's just a matter of exposing that to the evaluation function, but other evaluation functions might work as well, or even better.
If it doesn't evolve into a corner, it probably could master those games given enough time, but how much is enough time?
Deep Q Networks (the Atari AI you're thinking of) and really all reinforcement learning systems generally fail to make it past even the first screen.
I believe some researchers have tried this approach and had some success, but I'd like to see how far DeepMind (for example) could take it.