On: Ask HN: How does LLM serving infra work?

[ActorNightly]

Generally, having to unload data from GPU ram, and load a new set of weights in is quite expensive, so my guess is that the backend is built out where an input gets reservation to some cluster based on some ordering, and the batch is ran through.