The discussion of "memory storage" is weird. You can't magically make ChatGPT remember more from before the context window, and I'm not sure how that would reduce costs.
Perhaps it's a free vector store for retrieval-augmented generation, where instead from pulling from knowledge base vectors it pulls from previous conversation bullet point vectors?
Altman mentioned[1][2] earlier that they were working on a "stateful" API for release this year.
> 2023: A stateful API — When you call the chat API today, you have to repeatedly pass through the same conversation history and pay for the same tokens again and again. In the future there will be a version of the API that remembers the conversation history.
Maybe it's an RAG-based thing, but that'd be underwhelming given the promise.
Wizard of Oz, or true magic?
(In the same interview, Altman also claimed progress toward releasing million-token context windows this year. Wowzers)
Perhaps it's a free vector store for retrieval-augmented generation, where instead from pulling from knowledge base vectors it pulls from previous conversation bullet point vectors?