As the other posters have said, size doesn't seem to correlate to anything. More than that I don't know.
That aside the point is there would be less nerve input for the brain to process from five grams of fat, five grams of skin, or say an organ like the eye. Let alone any other other details it must acount or provide for. Further complicating things is neural plasticity - if I recall correctly "optical" areas of the born blind tend to be repurposed for other senses as opposed to say mathematical ability.
TL;DR: There is evidence but certainly not good enough to tell us anything about sex differences in humans.