It appears that this is quite similar to Google's Depthwise Separable Convolution from 2019.
In a normal CNN network, the kernel has a spatial width and height and a certain number of input and output channels. Their MLP blocks are effectively a convolution where things are mixed up spatially (but channels remain the same) followed by a convolution where channels are mixed up (but only within one spatial block).
The obvious advantage of this architecture is that the weights for remixing channels can be shared regardless of where in the image it occurs. So one would expect this to be more translation-invariant than a typical CNN architecture. But for other tasks - such as image reconstruction - that loss of spatial information might actually hurt performance.
To me, this looks like an approach that isn't really that new and isn't performing better, but mainly it's just different from what people usually use.
In a normal CNN network, the kernel has a spatial width and height and a certain number of input and output channels. Their MLP blocks are effectively a convolution where things are mixed up spatially (but channels remain the same) followed by a convolution where channels are mixed up (but only within one spatial block).
The obvious advantage of this architecture is that the weights for remixing channels can be shared regardless of where in the image it occurs. So one would expect this to be more translation-invariant than a typical CNN architecture. But for other tasks - such as image reconstruction - that loss of spatial information might actually hurt performance.
To me, this looks like an approach that isn't really that new and isn't performing better, but mainly it's just different from what people usually use.