As an example, what I tried right now: USER I'm responding to you roughly every ...

Retric · 2024-01-25T16:18:05 1706199485

That’s almost but not actually waiting. Picking the nest token for a context window is something GPT-4 is quite good at. But the underlying approach has some real limitations which I discussed in response to your other comment.

Using 1 hour instead of 10 seconds and then list every Current Time incrementing by .01 seconds and you overflow the context window. The solution isn’t to have a larger context window, the solution is to have an internal representation independent of that context window.

The issue is you can’t train a system to use that internal memory by simply giving it a large corpus of text examples. The text lacks the internal representation associated with its creation.

mewpmewp2 · 2024-01-25T17:00:45 1706202045

In this case you wouldn't spam the context window, you would perhaps just send latest time every time and lose portion of the previous times.

Retric · 2024-01-25T21:00:48 1706216448

That solves this specific issue, but it isn’t a general solution to the finite size of a context window. Ideally the system should be able to decide for itself what information it needs to pay attention to, but LLM’s simply don’t operate in that fashion.

mewpmewp2 · 2024-01-25T21:55:41 1706219741

But you can, e.g. you allow GPT to call functions to store embeddings that it can later retrieve for the memory. Meanwhile default context memory window is just recent info more detailed and historical info gets more and more summarized, possibly summarized by GPT itself.