Arguments by symmetry are incredibly powerful and useful. Why do you think they don't get to the heart of things? (Granted, X must be chosen so that it is a fair comparison, but when it's not, that choice is what should be attacked, not the structure of the argument.)
An argument by symmetry is that something shouldn't change when some irrelevant bit is changed -- such as the identity of the actors. Yes, the preconditions for using "If I X then Y" when trying to argue that "when someone else X', then Y' " is wrong are that X' is relevantly similar enough to X, and Y' is distinctly different than Y.
