Context matters some. But it's not like people's opinions reverse when they walk into the office.
If he is biased against gays, a concern raised by helping fund the removal of one of their civil rights, people can be reasonably afraid that this will eventually affect them in the workplace.
It need not be something obvious, either. Most decisions made by CEOs, especially decisions on who to reward and promote, are almost entirely judgment calls. Do you want to work for a guy who might not treat you fairly? "Yes" and "no" are both reasonable answers.