On: Transformers Can Do Arithmetic with the Right Embeddings

[Havoc]

I'd say the key point here isn't that they "need" specialised embeddings, but rather that it improves things and it can samewhat manage without.

That's a far more surmountable problem. Maybe you need one model for biology and another for coding etc. i.e. Broad split by domain. Still weak AI not true general in AGI sense, but still seems like a good next step