It almost seems like animal classism. I feel the same way when people call the cops on their neighbors for leaving a dog outside in a yard all day, but express no moral outrage at the plight of stray dogs.
Certainly not. A wealthy slaveholder could feel sympathy with the slaves and not want them beaten unnecessarily, and care more about the good treatment of the slaves of neighbors than to those held on another land and continent.
The upper-class homes of 1800s Britain had many working class servants. I have no doubt that in general there was some sympathy more for one's own servants over those of a neighbor's, and even more than for British landowners in Ireland.
That is, classism exists even in the presence of "the innocent and natural bias toward sympathy", because there's nothing in sympathy which demands the object of sympathy be treated as an equal.