The unique value of inertial navigation is that it requires neither receiving nor sending a signal that can be jammed or spoofed, hence why the military uses it for everything.
Citation: Myself, I worked on a variety of systems including INS and GPS in the military.
LR gyros are also limited by physics to a certain limit of precision, and this is the specific mechanism by which they can drift (from what I understand though building larger gyros might help). So flying for a long time.... I don't know to what extent we know how much drift might occur. It would certainly be possible to build in some checks here though since landing in Iran instead of Afghanistan would seem to show some discrepancy between the two systems.
I am just not sure you can be sure that relative position will be as accurate as absolute position over an extended period of airtime.
I'm am pretty sure this is very wrong, although I'm not an expert enough to downvote you. Having multiple systems allows one to average out (statistically independent) drifts; it's not just a protection from failure. I don't think there's anything special about a "slow turn" which will cause military-quality INSs to have correlated drifts.
I think the real problem is that INSs are simply to bulky and heavy for drones, rather than anything to do with drift.
I heard some Russian ICBMs had a special window so that cameras can see constellations for that purpose.