On: Were RNNs all we needed?

[ithkuil]

the distinction is fuzzy indeed, especially if any thing that you "program in manually" has some parameters that are learned.

Conceptually we already have parts of the model that are not learned: the architecture of the model itself.