Why not? I'd argue a racist or misogynist would make it worse for society than a pedophile (mainly because I think there is a very strong notion of protecting kids in society, but a less stronger notion when it comes to minority groups or women).
Do you think a president would make bad policy choices because they are secretly attracted to minors?
Do you think a president would make bad policy choices because they are secretly attracted to minors?