On: Trained KV cache bank turns any LLM into Jev like Model

[faangguyindia]

Here you can see a 26B model hacked using a trained KV cache bank, responding like Gemma. The latency I am seeing is <127 ms.