My occupation puts me in positions of authority over strangers from time to time. I pretty much always default to '"sir" and "m'am" in those cases.
"You" would seem to be more appropriate.
To add a little more context, I work in emergency services (fire/EMS), so if I'm addressing someone, it's typically to give them instruction to keep them safe ("Sir, I need you to ______"). I also use 'sir' before I know someone's name ("Excuse me sir, could I get your name?").
The word can certainly have different connotations depending on the relationship. A service employee would generally refer to a customer/employer as "sir," and I suppose there's an air of subservience there.
On reflection, I suppose you could call a firefighter a "service employee," and civilians on an emergency scene are in many ways "customers," so my use of the term might just be an extension of that relationship...
In other English-speaking countries I've been to, "sir" is a genuine sign of subservience.
Personally the situation where I'm most likely to call someone sir is where I'm giving them a forceful verbal put down. "With due respect sir that sort of language is unacceptable here." or what have you.
I'm in the UK so also when addressing a Knight of the realm in a formal setting.
(And similarly with the leafletting, cashiering, tour-guiding and door-holding occupations I've had in the past. You want people to feel important, and you're in a subservient position, so you elevate them verbally. Basically, when you're a Baldrick.)