The question should really be the other way around - LLMs are just computer programs, and in general, computer programs are theoretically deterministic. Given the same inputs and the same initial state, they produce the same outputs. So the question is, what would make an LLM non-deterministic?
For any computer program, there are two main ways determinism can be lost:
1. Randomization is introduced deliberately, e.g. via random number generators.
2. The program's behavior depends on factors beyond its explicit inputs, e.g. its execution environment, timing, concurrency, or interactions with other systems.
Actual deployed LLMs typically lose determinism because of both of the above factors:
1. On each request, the core model outputs a list of likely tokens, ranked in probability order. To avoid repetitiveness or getting "stuck", particularly in natural language output, it's common to introduce randomization when picking a token from this list. This is what's called "temperature". This is deliberate non-determinism that's applied to the output of the core neural network model. Setting temperature to zero can eliminate this kind of non-determinism.
2. In practice, consumer-facing deployments run in complex hardware environments - distributed processing, GPUs, hardware shared with other processes, etc. This leads to non-determinism due to things like concurrency and timing issues. Calculations can execute in different orders, numerical errors can accumulate in different ways, etc.
For companies like OpenAI or Anthropic, which serve millions of concurrent sessions, eliminating non-determinism from #2 would be difficult, and not worth the effort.
But if you're running an LLM on e.g. a single machine, it's certainly possible to eliminate non-determinism. People do this for various kinds of models. Machine learning frameworks have explicit options to help with this. E.g. with Pytorch, you can use `use_deterministic_algorithms(True)` to force it to use deterministic algorithms where possible.
One point that sometimes confuses people here is that even if you run a perfectly deterministic LLM, the output is unpredictable in general. That's because the output of the model depends on a large number of trained weights, and the only practical way to "predict" what output a model will produce is to run input through the model. This is not non-determinism, though.
In diagram terms, at a high level an LLM looks like this: input --> neural network --> token selection --> output
Token selection can be randomized, but doesn't have to be. When it's not, the above is perfectly deterministic in theory, and can also be run that way in practice.
Now take that diagram and run it across a data center full of machines, and achieving determinism gets more difficult. It could be done if we wanted to, but in general it's not important enough to try in that scenario.
thank you for the detailed answer, this is all very interesting to me as I'm just starting a Masters in AI. I always had the wrong idea that LLMs couldnt be made deterministic. Now that I know they could be this opens up more possibilities for me.
Glad it was helpful. If you have time for some extracurricular study, I'd highly recommend implementing an LLM. Implementing a toy version is really not that difficult, and can be very illuminating.
There are quite a few resources on the web that can help with this. One good one is Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out.": https://www.youtube.com/watch?v=kCc8FmEb1nY
Two separate things. Sampling randomness goes away with greedy decoding: always take the highest-probability token instead of sampling, which is what temperature 0 approximates.
The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.
