This is not accurate. My credit score is quite low for a person with zero debt and great financial standing, because I don't have a credit card. I am being punished for not buying a product that I have no need for.
Simply having a credit card doesn't cost anything. You're choosing to live your life in a way that is contrary to what our society has determined is a positive financial profile. If you're trying to make some kind of point, or protest something, fine -- but then don't complain about being "punished."
When was the vote / referendum on this where "society" made this determination?
At most, a few credit ratings company made this decision and "society" tolerates it for convenience. An equally workable system would be to to default to a high score and deduct points for bad behavior, but this would be bad for business as it wouldn't force people to get credit they don't need.