This phrase suggests that ALL speech that non-absolutists dislike will be categorized as harmful enough to society to be disallowed. That is a view nobody I've ever met holds.
I am not a free speech absolutist. I believe 99.9999% of speech I dislike, even speech I dislike intensely and believe is harmful to society, should still be allowed by society. That I think 0.00001% crosses a line where its harm outweighs the benefit of an zero-tolerance, non-negotiable absolute freedom of speech.
Throwing off a facile "oh, you'll just ban speech you don't like" is just jingoism. It's no different than saying people who favour less immigration are Nazis, or people who favour universal health care are communists.
It avoids thinking by name-calling.
Ordering someone to shoot you? Aka "clear and imminent threat" ?