I'm not sure how that would work. You'd need to be able to reverse the operations in the KV cache, and I'm not sure if it's doable or not. And then you'd need to pick another path, or "rebalance" the probabilities or you'd go the same token path. CoT is probably easier to implement (but not necessarily better).
Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints :
- use checkpoints to save KV cache before trigger CoT
- trigger CoT, save result as a summary
- go back to previous checkpoint
- instead of generating tokens, add result of CoT summary as input tokens
- continue normally
For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc.
This idea is free.
I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.
