I thought it would be fun to have a fully AI twitch streamer that could interact with chat while playing a game. Since Colossal Cave Adventure is simple to parse on the console, I hooked it up over the last week.
This was pretty interesting to set up! My main takeaways are:
1) ChatGPT (3.5-turbo) is very bad at the game, and it makes the same mistakes over and over, even when it has an example of the same mistake (and correction) in its recent context. The illusion of this type of AI feeling "human" fell apart pretty quickly for me.
2) At the same time, being able to give instructions to something in natural language and get arbitrary working JSON back is awesome, and really opens up a lot of possibilities for quick-and-dirty solutions (albeit via an expensive API). Even if you get malformed JSON back, you can literally just say "this didn't parse, fix it," and 9 times out of 10 it does. It's also impressive that it can do two things at once - talk to chat and play the game (usually - sometimes it decides to write its chat responses into the game).
3) I think a big issue for chatbots is that ChatGPT seems to learn from its own bad behavior. It's easy to get long streams of nearly identical responses from it, since if you feed its own responses back to it in the chatlog, it seems to use them as examples of how to act in the future. But if you omit its own responses, it doesn't have any context for what it has already tried. Maybe someone else has found away to get around that?
I'm not sure how much further I could take this, but hopefully it's interesting to someone. I don't think it has much of a chance of beating the game without a lot of handholding from chat. It's maybe one step up from Twitch plays Pokemon, honestly. When I get access to GPT-4 I'll run it again and see if it improves at the game.
