On: Unit Economics of LLM APIs

[danielhanchen]

Ye the article is pretty interesting - I was trying to calculate inference provider's profit margins for say Llama-3.1 8b including electricity costs and the power consumption of GPUs, and I seem to always get to around $0.20/1M tokens as the minimum cost if you amortize a GPU's cost over 1 GPU. If you amortize it over 2 years, then maybe $0.15/1M tokens.

I'm very intrigued about OpenAI's profit margins, if they are indeed true.

[htrp → danielhanchen]

4o is heavily quantized perhaps and runs on hardware that conceivably could be used for 3-4 years?

[danielhanchen → htrp]

Fair point! Ye if it's 3-4 year amortization that's reasonable - quantization could be another reason why some benchmarks for 4o aren't that good vs pure float8 / float16, and also how they made it much faster