I honestly understand that quite fine. The GP of my original comment stated "You say that as if letting someone's gender affect your decisions is somehow, magically, inherently always ALWAYS wrong. It isn't." It seems the three of us are in agreement that in the lavatory case letting gender affect decisions (in this case the common decision for one gender only facilities) is not wrong or sexist. Yet you jumped on him/her as a defender of sexism and prejudice when in reality you, and probably everybody else, takes gender into account in many benign or benevolent contexts as well.
Different toilets are accommodating for different physical differences. It's not racism if someone is selling suntan lotion on the beach and only approaching light skinned people.
In the case of poverty, this is not a genetic difference - it's the product of culture. Poverty has no color, anyone can be affected by it. Why then, when the problem is not of skin color but of poverty, are we talking about skin color at all? The way I see it is that there are people who need a hand up because they lack the resources to get up - I don't see a whole race as being disadvantaged because that is extremely presumptuous and inaccurate - there are plenty of rich black people. The simple solution to this is to make things like education free if you earn below a certain amount. This resolves the issue that affirmative action tries to tackle without siding with a particular race. It's just common sense.
My point, and that of the previous commenter, is that taking gender into account when making decisions is not always sexist. And I am disputing your assertion that this view automatically makes one a defender of sexism and prejudice. Separate male/female lavatories is one example of accounting for gender in a non-sexist manner. Someone else mentioned organizing hotel accommodations with same gender persons sharing rooms. Department stores put men's and women's clothing in separate areas.
I don't see how poverty, affirmative action, or race is relevant so I have nothing to say on that.
