I agree that there is a middle ground - when someone's quality of life is seriously suffering because, for example, someone is shouting abuse at them all day then something must be done. However, taking serious offence at people's ignorant opinions or ways of expression is unwise and in itself ignorant.
I'm anticipating prejudice so I shall address it pre-emptively - I am, in fact, a minority. In many places on this planet, I risk serious injury just for being what I am. People have expressed extremely ignorant opinions about the group I fall under in workplaces. I initially took offence upon hearing what I heard but then I moved past that stage and accepted that one cannot understand something they haven't dealt with directly. They were not trying to be mean, they merely did not understand because of their limited experience. How can I take offence when this is the case?
That's certainly a good sign that social sanctions for sexism, racism, etc work. People ~should~ think before they speak, much more than they do in most sites and blogs.