EDIT: Just to add that your comment's two halves are not logically connected. Even if we do act in this way, it doesn't make it a "critical part of what we are", or if it does, there is no reason that "critical parts" cannot be altered if need be. In the early 19th century most people would have thought that a 10 hour working day was too liberal - an 8 hour working day would have been unthinkable; but that "critical part" of people's way of perceiving society was not set in stone, as it turned out.
Whether this all derives from biology is hard to say - however the alternative is to suggest that we are all somehow indoctrinated to value physical attractiveness and would otherwise be saintlike in ignoring things that suggest genetic unfitness, such as obesity, small stature, disfigurement etc. Look at the animal kingdom - I don't think animals are gauging partners on their ability to tell a funny joke, or having the correct politics. The sex drive is one of our most basic drives. To think it's not still driven by basic prerogatives does not seem reasonable to me.