Quaternions are used to describe a rotation in 3D-space. Three numbers give the rotation axis and the magnitude of all four is the rotation angle. They are used instead of euler angles because they don't have a singularity problem (gimbal lock). I don't know if this plays a role here. Could these neural networks represent 3D transformations really well?
Geoffrey Hinton's idea of "capsules", which I don't really know anything concrete about, tries to address the recognition of objects subject to rotations, etc. That's a topological/structural strategy within the neural network, though, so quite removed from an idea like quaternions.
It's worth observing that what distinguishes complex numbers from 2-vectors like (x,y) is that there's a multiplication rule that corresponds to rotation around the origin. Similarly with quaternions. But you can also just use them as glorified vectors of 2 or 4 elements.
