My dad explained what was wrong and I was horrified but my sister was fascinated. This has bothered me my whole life.
Really think about the kinds of questions you're asking yourself as you're looking at this painting.
Now, imagine seeing a large, stately portrait of a well-dressed woman, around the same older age.
What do you notice first about her?
For me, the answers are:
Posture (confident? not?), eyes, watch, spectacles (was he a professor? scientist? businessman?) What did he do to deserve a painting?
Dress, beauty (or lack thereof - is she beautiful? wrinkled? old?), hair. Who was she [related to] to deserve a painting?
Did you ask yourself what the woman might have liked to do?