I think it's often hard to tell. One way of addressing this problem is to eliminate the friction associated with switching roles.
We always pair with two keyboards, two mice, and two monitors plugged into the same machine. There's just too much friction associated with switching spots if you only have one keyboard.
If you don't have the equipment for that setup, you can set up a virtual pairing station with ssh and tmux pretty easily.
so basically both are fiddling around with the same cursor?
But yeah, 2 keyboards are a necessity