I think its much easier in a space where you already have a precisely measured framework.
Disclaimer: I haven't prototyped this.
I think with a good enough continuous camera movement you can track from a zoomed out or slow-pan view that takes in multiple registration points (e.g. room corners) to a zoomed in view of a detail of interest keeping track of corresponding details. That lets you map back from the zoomed out view where you could correspond to the recorded data by the visual registration points.
