Each person contributes to the flavor of the community in which they participate. It is within the right of those who lead the community to curate its flavor by adding or removing people when necessary.
In this his case, his removal was well-justified for the cause of keeping the community welcoming and safe-feeling for members of all genders.
Also don't be surprised if lots of disparate organizations agree that certain behaviors are inappropriate.
RMS and others are free to say or do whatever they please, but they aren't free from experiencing the consequences (social or otherwise) of their choices.
In fact, your message here makes me feel unsafe right now. Something needs to be done, call the mods.