Nothing wrong, you can organize your own conferences in whatever way you like.
But if you act morally outraged at what others do in the conferences they organize, don't expect much sympathy.
... and you can lose out on the participation of a valuable subset of contributors. This isn't a moral issue, it's a question of maturity, decorum, and being inclusive.
Likewise, if you're a socially ignorant asshat, don't expect people to like you.
Or come to your conferences.
Or invite you to speak at theirs.