On: The shrinking landscape of linguistic diversity in the age of LLMs

[BoredomIsFun]

> Try tinkering with a base model and you'll be surprised how diverse it is.

Have you tried? I have. Not much different from RLHFed; full of tics and slop, similar but slightly different from intsruction posttrains.

[orbital-decay → BoredomIsFun]

Yes, although admittedly I haven't tried recent ones which have a bunch of synthetic data in them (mainly to aid the reasoning), and are usually only available after mid-training. One look at the logits/output distribution and it's clear the base is pretty different.