We have to take it as far as emphathy combined with reason (which extends our ability to empathize) will take us, precisely because this is the only self-consistent approach other than sociopathy. If you're advocating the latter, then I'll admit that it is also fully self-consistent, and rejecting it is really just an ethical axiom that I subscribe to.
How much further do we have to take it? What determines this limit?
You might not find this as self-consistent as you think.