I mean the whole way looking at it seems wrong to me. There is no "rest of the world" in relativity. Assigning some objective vector to everybody doesn't work. These only make sense from some specific point of view.
By the question you asked now I'm assuming, you meant "but hey, without GR..", but even without GR, ignoring that the universe is expanding, assuming flat space time etc. If the universe consisted just of 3 bodies, 1 being you and 2 being rest of the world, then the way of thinking you described still doesn't make sense in context of relativity and may lead to some confusion (apart from it being, to me, incoherent in context of special relativity).
But maybe I'm missing something from your picture, I'm happy to read and learn.
You can certainly imagine a scenario where two objects measure their mutual distance as being constant in time. You can also imagine other scenarios, but I'm asking to imagine the scenario where two objects do measure their distance to be D and then measure it again and it's still D, and measure it again and again and it's still D. That's just the definition of standing still with respect of each other, and when you plot a space time diagram, their space curve is parallel (because their spacial distance doesn't change).
Special relativity doesn't make that scenario impossible. It doesn't force things to move. It just describes what happens when things do move (through spacetime).
Special relativity definitely won't be a problem. Everyone see timings and speeds a little differently, but everyone can also do the calculation from anyone else's point of view. Objects that are in the same reference frame are objectively in the same reference frame in special relativity; everyone agrees. The reference frame is determined only by velocity. And the distortion is determined only by the observer's velocity, so both objects will have the same distortion.
