On: Don't trust large context windows

[petesergeant]

Is there any chance that this is because training corpus largely consists of documents shorter than the advertised context windows?