On: Prompt caching for cheaper LLM tokens

[moebrowne]

For ChatGPT:

> Prompt caches are not shared between organizations. Only members of the same organization can access caches of identical prompts.

https://platform.openai.com/docs/guides/prompt-caching#frequ...

[maxloh → moebrowne]

I don't find it really viable. There are so many ways to express the same question, and context does matter: the same prompt becomes irrelevant if the previous prompts or LLM responses differ.

With the cache limited to the same organization, the chances of it actually being reused would be extremely low.

[qeternity → maxloh]

In a chat setting you hit the cache every time you add a new prompt: all historical question/answer pairs are part of the context and don’t need to be prefilled again.

On the API side imagine you are doing document processing and have a 50k token instruction prompt that you reuse for every document.

It’s extremely viable and used all the time.

[jonhohle → qeternity]

I’m shocked that this hasn’t been a thing from the start. That seems like table stakes for automating repetitive tasks.

[qeternity → jonhohle]

It has been a thing. In a single request, this same cache is reused for each forward pass.

It took a while for companies to start metering it and charging accordingly.

Also companies invested in hierarchical caches that allow longer term and cross cluster caching.

[IanCal → maxloh]

It gets used massively in a conversation, also anything that has a lot of explain actions in the system prompt means you have a large matching prefix.

[babelfish → maxloh]

Think of it as a very useful prefix match. If all of your threads start with the same system prompt, you will reap benefits from prompt caching.