You're assuming most people can afford to martyr themselves to pay extra on every purchase out of a moral stance.
Even if someone can afford that, I think it's been demonstrated futile to try to affect market behavior on the basis of employee treatment that's so normalized to everyone. Boycotts have worked on other specific issues on certain brands but it seems like people just don't care enough about Amazon's worker treatment or Nestle's horrible activities in Africa. So if those are never going to change without legislation then I can't really hold it against anyone for not paying out the nose to avoid Wal Mart and Amazon.
