... and you can lose out on the participation of a valuable subset of contributors. This isn't a moral issue, it's a question of maturity, decorum, and being inclusive.
But if you act morally outraged at what others do in the conferences they organize, don't expect much sympathy.
Likewise, if you're a socially ignorant asshat, don't expect people to like you.
Or come to your conferences.
Or invite you to speak at theirs.