A related questions: How do LLMs format code so consistently? I mean, when you write middle-indented things like struct fields in Go, how do they know in advance what the largest field name will be?
Two mechanisms, running backwards and forwards through time.
First, LLMs can actually plan ahead - to a limited degree. Counterintuitive but true. So by the time the indentation is emitted, an LLM can already have something of a clue as to what the field names may be, and pick the indentation length accordingly.
Second, all LLMs want to conform to their context - which, in generation, includes their own past choices! This "consistency drive" is an innate instinct, originating at base model level, and it's one of the strongest and most conserved behaviors across all LLMs.
When an LLM sees the indentation length, it will try to pick the variable names that would conform to it.
I'm sure that you can actually find or craft some corner cases, in which both of those things would fail to "meet in the middle", and inconsistent indentation will be emitted. But it usually works well enough.
> First, LLMs can actually plan ahead - to a limited degree. Counterintuitive but true. So by the time the indentation is emitted, an LLM can already have something of a clue as to what the field names may be, and pick the indentation length accordingly.
I think this is a bad explanation. It's like saying that if execution enters a function in your program, the program has planned ahead because the rest of the function exists.
The LLM has circuits/basins which are ~guaranteed to emit a certain longer answer once inference has entered them. This is why it's capable of forming words in the first place.
