Just Ask for Generalization (2021)

[jxmorris12]

https://evjang.com/2021/10/23/generalization.html

[xg15]

(2021), still very interesting. Especially the "post-overfitting" training strategy is unexpected.

[dev_hugepages → xg15]

This is talking about the double descent phenomenon (https://en.wikipedia.org/wiki/Double_descent)

[luckystarr → xg15]

I remember vaguely that this was observed when training GPT-3 (probably?) as well. Just trained on and on, and the error went up and then down again. Like a phase transition in the model.

[esafak]

The low sample efficiency of RL is well explained.