All that fuzz and in the last figure showing their touted great invention being beaten by vanilla Transformers, when their invention takes more parameters, more FLOPs, and is not parallelizable. Did I miss something or is this absolutely useless?
Well it’s fascinating /if true/, but it’s not really proven and that’s why the experiments are so disappointing to me. They put forward this grand theory and then are entirely unable to capitalize on it or make some kind of useful prediction from it.
