On: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

[gundmc]

Well, their huge GPU clusters have "insane VRAM". Once you can actually load the model without offloading, inference isn't all that computationally expensive for the most part.