It's more complicated than that. Victorian morality was also about (for example) discouraging child prostitution, of which Victorian society had quite a lot. Middle-class moralizers eventually morphed into social reformers when they discovered it was hard to be a good person in tough social conditions. And so on. The good and bad are not very easy to disentangle.
A related element is that apparently there was a large surplus of women at the time. The imbalance contributed to an increase prostitution and women getting stuck in unhappy marriages.