These glitch tokens are all near the centroid of the token embedding space. That means that the model cannot really differentiate between these tokens and the others equally near the center of the embedding space, and therefore when asked to 'repeat' them, gets the wrong one.
That happened because the tokens were on the internet many millions of times (the davidjl user has 163,000 posts on reddit simply counting increasing numbers), yet the tokens themselves were never hard to predict (and therefore while training, the gradients became nearly zero, and the embedding vectors decayed to zero, which some optimizers will do when normalizing weights).
I think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance.
Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...
This comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)
That happened because the tokens were on the internet many millions of times (the davidjl user has 163,000 posts on reddit simply counting increasing numbers), yet the tokens themselves were never hard to predict (and therefore while training, the gradients became nearly zero, and the embedding vectors decayed to zero, which some optimizers will do when normalizing weights).