The most interesting thing I've found with wearing better quality clothing is that people notice, but don't realize this is what they're noticing. I get comments about my shirts from clients and when I go out in public random people will stop me to say something. Nobody says something about some technical factor of the clothing itself, but they will say things like "that's a nice shirt" or "wow, that shirt is really blue" (better quality cloth tends to have more consistent and lasting color as well).
This is one of those areas where quality isn't just expensive, it's more that quality only comes with being able to judge with expertise. Like most, I am unlikely to ever become an expert in textiles, so I found that having a reputable tailor do work for you was the best way to dress well without breaking the bank. My clothing lasts longer, looks better, and most importantly it feels better, all for a 20-30% markup over going to the mall.