But that is not what "there is no physical reality to the matter" means, if one chooses not to use any relative point there is still an objective distance between each object with respect to every other object and accelerations for each of those distances.
"The matter" in that quote is the matter of whether the Earth orbits the sun or vice versa. Naturally there's observable physical reality to distances and whatnot, but if you use the laws of physics to make predictions about those observables, the math works regardless of which body you assume the others revolve around.
