Do you have a citation re:mathematics? As a person in mathematical physics, I would be /very/ surprised to hear that math is a more gender balanced field than physics.
I do somewhere; I'll track it down and relay the numbers. But also I'll say that this matches my own personal experience: go to an academic cryptography conference sometime and note how many more women you see than in a generic CS conference. There are more women mathematicians than computer scientists and physicists.
I haven't attended any crypto conferences but, I do work a bit with high school STEM programs and I can tell you.. higher ratio of male to female, BUT, those female students are far more interested in the math / crypto fields than the males (which leads me to believe they will stick to it career wise vs the male students. So it's not too surprising your observations.