There is a middle ground between "sending people to camps for a wrong joke" and "no one should be held accountable for any of their actions if they claim they are jokes"
There is no yet identified metaphysical difference between speech and other actions, and freedom of expression laws frequently also protect non-speech actions such as burning flags.