Character.ai shows how to reduce LLM costs by 13 times

[Oksitaine]

https://blog.character.ai/optimizing-ai-inference-at-character-ai/

[gnabgib]

Article title: Optimizing AI Inference

Discussion (63 points, 14 days ago) https://news.ycombinator.com/item?id=40739225

[uberman]

If I reduce the cost of something by 1x it becomes free...

[Oksitaine → uberman]

Yes it's true, i mean he LLM costs for Character.ai are 13 times cheaper (thanks to their optimization with KV cache) than if they had simply used the API of an LLM service like Anthropic or ChatGPT.

[uberman → Oksitaine]

I think it is a great optimization. My point is that they can reduce the cost by 92% which is less than 1x (a fantastic optimization) but if they reduce the cost by 1x (100%) it would mean that the service would be free. Reducing the cost by 13x means that for every dollar I spend they would give me 13 dollars. This is a not uncommon error and i would not mention it, but it is the direct claim in the heading.