Positional description matters for transformers arithmetic

[Handy-Man]

https://arxiv.org/abs/2311.14737

[thomasahle]

Also positional encodings. A few months ago I did a bunch of experiments with length generalization in addition and multiplication (https://github.com/thomasahle/arithmetic-transformer).

It turned out that not using positional encodings at all (relying entirely on the causality mapping in the attention) did better than all other methods like Learned PoE, Sinusoidal, RoPE and using an LSTM below the transformer.

[Kerbonut → thomasahle]

Can that be part of a multimodal language model to better understand and perform arithmetic operations, enhancing the LLMs numerical reasoning?

[inciampati → thomasahle]

Fascinating, thanks for sharing this!