Another interesting run during the TASBot block of AGDQ was the Brain Age run: https://www.youtube.com/watch?v=wRc5RH4tR1E (you can see pictures of the setup in the article)
In the run, the TASBot is hooked up to the DS's touch screen...and hilarity ensues. (it's a throwback to a Brain Age speedrun with a similar gimmick)
I was wondering how it tricked the game into recognizing the correct answer. Does it end each question by writing the correct answer inside an already darkened portion maybe?
The tracking pays more attention to the length and position of your strokes, rather than doing what you might think of as OCR on the finished characters.
Here's a tool-assised run that the TASBot run was clearly inspired by: https://www.youtube.com/watch?v=_p_UtfOmp9E - you'll notice there's no opportunity for the actual numbers to be drawn in many instances.
For the most part, the images were selected/drawn in a way that fools the engine. The team responsible for the speedrun made a brief writeup that explains some of the stuff they ended up using: http://tasvideos.org/GameResources/DS/BrainAgeTrainYourBrain...
It's consistently amazing how ingenious some of these runners are in figuring out ways to break out of the apparent rules of a formal system and get creative with it. The TASbot guys have also done things like RAM manipulation of Pokemon games in order to rewrite the game code into displaying twitch chat.
AGDQ and SGDQ are two of the coolest video game-related events held every year. They're always streamed on Twitch, and recordings are posted to youtube. Anyone who has even a mild interest in gaming should check them out. https://gamesdonequick.com/
I watched a few speed runs from last week's Awesome Games Done Quick marathon and several of the runners were able to perform some impressive game breaking exploits in real-time. They didn't always manage it, but for most of the finicky exploits they usually had 3-4 runners doing the same trick simultaneously and at least one of them got it.
In particular, two out of four runners of Super Mario World managed to move objects to exact pixel positions such that their x/y coordinates represented machine code to jump to the credits sequence, and then get the game to jump to that code.
Many of the exploits done at AGDQ require "frame perfect" timing. Depending on the game, this is 1/24th of a second at best, and 1/60th of a second at worst.
You may also enjoy this speedrun of pokemon blue from last year's Summer Games Done Quick, where all 151 monsters are caught as quickly as possible. Essentially nothing is done correctly: menus are underflowed, pointers are gymnastically redirected, PRNGs are manipulated, various state machines are thrown out of alignment and fun is had by all.
The amazing thing is that this isn't a ROM hack, it's all done by exploiting glitches and completely hijacking the game. And it's done in real time on real hardware. The stream is worth watching.
I guess you only watched the video. If you scroll down a bit further it reads:
> I thought it'd be fun to pretend that SMB3's legendary lead developer, Shigeru Miyamoto, had left an intentional "back door" hidden in the game which we had only just now discovered.
In the run, the TASBot is hooked up to the DS's touch screen...and hilarity ensues. (it's a throwback to a Brain Age speedrun with a similar gimmick)