I thought the reason these guidelines are set is so that everyone is protected--attendees feel comfortable, and organizers don't have to feel like they're on their own with making decisions on things that are making everyone(potentially) uncomfortable. Conference organizers certainly don't enjoy confrontation.
Also, thanks for the condescension behind the girl-elevator scenario. That really strengthens the argument. Ladies don't report conversations in an elevator. They report feeling unsafe.