Let's say the baker was a victim of sexual abuse and the client was asking to bake a cake rape-themed. (Not in support, you can suppose it was a replica for a documentary let's say). The baker refuses, the client sues, who is the first intolerant?
Another case, a client enter the shop of a muslim baker and asks for a depiction of Allah on a cake. Who is the first intolerant?
Maybe you have answer for all such situations, but how confident are you that a majority agrees in all cases?
He only objected to a custom message defying his belief. There are good arguments on both sides. (my stance is sort of about scale as in if he had enough employees to delegate or he was the only cake-maker there)