This is exactly why we need free speech. It's a pragmatic position. Of course I would like to live in a world where people don't spew vitriol. But for the reasons you outline that cannot happen except in an authoritarian society.
As an illustrative point both Russia and China have constitutionally protected freedom of speech. They just added enough caveats that it's worthless.
I would point out that governments in authoritarian societies have a habit of labelling certain groups of people as enemies (either because those groups challenge them politically, or because it's convenient to have a scapegoat to blame the government's own failures on), so in practice you end up with just as much, if not more, vitriol spewed in authoritarian societies.
As an illustrative point both Russia and China have constitutionally protected freedom of speech. They just added enough caveats that it's worthless.