On: Grokfast: Accelerated Grokking by Amplifying Slow Gradients

[whimsicalism]

grokking doesn't and will not have practical uses, imo - it is just an experiment that revealed cool things that we mostly already suspected about implicit regularization

however, techniques we learn from grokking about implicit regularization might be helpful for the training regimes we actually use

[naasking → whimsicalism]

> grokking doesn't and will not have practical uses, imo

I'm not so sure. Reasoning is the next big hurdle, and grokking and parametric memory seem very effective here.

[1] https://arxiv.org/abs/2405.15071