OpenAI API Batch Requests

[davidbarker]

https://platform.openai.com/docs/api-reference/batch

[serjester]

This seems like a fantastic approach to smoothing their GPU utilization.

It’s so difficult to compete with OpenAI - they ship so many “no brainer” features that are constantly setting the bar for what an LLM provider needs to provide - chat API, multi modality, function calling, json output, it’s endless.

[dosinga]

Is running things in batch mode also cheaper? I don't see it, but it would be a nice way for OpenAI to smooth out their demand

[davidbarker → dosinga]

According to their tweet, the discount on API costs will be 50%, so pretty significant.

https://x.com/openaidevs/status/1779922566091522492