I'm kind of in the same boat. I never cared about fashion much. It always seemed hollow and vapid to me. Even as a kid/teen-ager/20s when it's supposed to matter the most.
Perhaps your circle of friends didn't care either.
For kids/teenagers/early-20s, what matters is belonging - observable patterns of behavior that follow are whatever gets amplified when everyone looks at everyone else in their vicinity for signals of approval.
FWIW, I also didn't care about fashion, and neither did my friends. I expect this to be the case with quite a lot of HNers - this comes from being a nerd and having nerd friends. We amplified each other on different patterns of behavior.
