One solution might be to put inside out camera trackers into the controllers as well, so they can judge where they are in the space, and even look back and scan your body position. possibly bandwidth and power limitations in wireless handheld controllers would make that a bad idea?
