Both men and women are owed the respect of not being offered frequent unwanted sexual requests. Of getting people to take them seriously, based purely on the merits of their work.
Both men and women aren't owed sex. And it's not a "downside" to not have something you're not entitled to not be given to you. It would be a downside to have to pay for clean drinkable water. Or not be allowed to get public transport to work.
If someone approaches you in public and asks you to have a coffee with them, you don't owe it to them to have it, and your refusal is completely ok. They have no right to get upset if you refuse. Just because everyone else has refused them before doesn't make them suddenly entitled to have coffee with you now.
If no one wants to have coffee with you, you need to make it appealing for them to have coffee with you. This isn't just looks, and you need to stop feeling sorry for yourself and blaming things you can't change.