A few years ago, I wouldn't have understood paying extra money for something that serves the same general purpose. What I didn't understand then was what you said now: It's all in the detail. I can buy less expensive clothing, but it doesn't feel as good and it doesn't look as good, and I think that the extra money spent is worth it for how it feels. Hell, you can even get a feel of it when you wear an American Apparel t-shirt versus a Cafepress one, and that's not a foray into fashion at all.
That's why it's fun for me right now, going to university with a bunch of fashionistas. Last year Abercrombie was everything; this year, the fashionable people are obsessive in their knowledge of clothing. And I love it. I don't know if I'll be able to afford their level of obsessive outfitting, but even if I can't I'm hoping that just being around stuff like that will rub off on me.