It shouldn't be that hard to imagine. If the objective is to predict human text, at some point the best way to predict text is to capture the kinds of knowledge that humans know and use in text generation. Human knowledge is a strong prior for human text generation, and so it shouldn't be surprising that for a large model its parameters would end up in a portion of parameter space that encodes some human knowledge.
By this token, the best way to learn Mandarin would be to start looking at text in Mandarin and trying to assign it meaning. While it theoretically can work, it is vastly more difficult than trying to relate it to a previously learned model. In fact, it is so difficult that we have never successfully understood a lost human language this way (though with the caveat that we don't have the huge corpus of text that GPT-3 was trained on in any of the lost languages we tried it on).
No, I said the best way to learn to predict human text is to capture human knowledge. I did not say how best to capture human knowledge. Obviously doing gradient descent over a huge text corpus is massively inefficient for a human. Whether its the best way for a machine to learn it is another matter. After all, the amount of data that humans experience through their senses during formative years, as well as the "computational work" that went into evolving human brain architectures surely dwarfs GPT-3's training regime.
