While I certainly wouldn't 'shrug off' anything, there is a very material difference between a privately held statue to a foreign historical figure with a ruinous effect on a large segment of the world's populations and a public commemoration of an actual traitor to the nation, responsible for attempting to defend the actual enslavement of a large population of Americans. Especially when the main utility of the former in this context appears to be to distract from the latter.
If it is a distraction to the latter, then shouldn't it come down? Next, you're going to defend the Gandhi statue on the Embarcadero in San Francisco, I suppose!