Not where I come from. I grew up in the area, not far from the first Trader Joes and down the street from the second, and it’s always been considered an upscale store because it had so many cheaper competitors a few miles away. It was founded to supply the wealthy neighborhoods of Pasadena and South Pasadena.
I think you are kind of both right. When TJ’S started they intentionally wouldn’t sell things like milk and eggs because they couldn’t compete with big chains. The niche they were after were “over educated and under paid”.
So they were famously cheap for things that poor and blue collar families weren’t looking for anyway. UMC goods on a LMC budget, really.