These people are real, and actually causing harm. Or do you argue that it is not the case?
The western Roman Empire fell apart a hundred years after Christianity became the state religion. Some historians blame tensions among christian schisms in Egypt/the Middle East for the byzantine empire losing those regions to the Arab invasion.
(All I’m saying with the above is that the justifications seemed plausible and reasonable to the educated people of the time. Read Pliny’s letter to Trajan seeking counsel for what to do about Christians, for example)
I think the answer is more speech, not less. Any exception you carve out will be abused in the future, based on the history of humankind and the behavior of governing entities throughout.
Take these exceptions to freedom of speech, add to them a codified framework for equity (which some are pushing for), and you're laying the groundwork for a society like that seen in Harrison Bergeron.