"Why is female->male fear different than other kinds of irrational fear?"
That's a loaded question. One of the core things to understand is that this is not an irrational fear at all. In fact, it is very rational.
If a person has had consistent, repeated experiences in which male conference attendees have exhibited unwelcome and aggressive behavior, then they are quite rational to expect that kind of behavior again. Public codes of conduct give people hope that when this kind of behavior does happen again, it will be dealt with appropriately.
It is wrong and irrational to expect harassing and aggressive behavior from everyone; that is stereotyping. But it is quite rational to expect the overall pattern to continue.