On: The FFT Strikes Back: An Efficient Alternative to Self-Attention

[porridgeraisin]

This idea exists (the broad field is called neural architecture search), although you have to parameterize it somehow to allow gradient descent to happen.

Here are examples:

https://arxiv.org/abs/2009.04759

https://arxiv.org/abs/1906.09529