"You" is pretty rude in America (or at least the North East).
To add a little more context, I work in emergency services (fire/EMS), so if I'm addressing someone, it's typically to give them instruction to keep them safe ("Sir, I need you to ______"). I also use 'sir' before I know someone's name ("Excuse me sir, could I get your name?").
The word can certainly have different connotations depending on the relationship. A service employee would generally refer to a customer/employer as "sir," and I suppose there's an air of subservience there.
On reflection, I suppose you could call a firefighter a "service employee," and civilians on an emergency scene are in many ways "customers," so my use of the term might just be an extension of that relationship...
Personally the situation where I'm most likely to call someone sir is where I'm giving them a forceful verbal put down. "With due respect sir that sort of language is unacceptable here." or what have you.
I'm in the UK so also when addressing a Knight of the realm in a formal setting.
Or when you want something from a stranger. "Excuse me sir, have you got any change..."
(And similarly with the leafletting, cashiering, tour-guiding and door-holding occupations I've had in the past. You want people to feel important, and you're in a subservient position, so you elevate them verbally. Basically, when you're a Baldrick.)
It's certainly less common than it used to be. My father -- born in 1927 -- typically uses sir and ma'am to people he doesn't know, unless he's being introduced, regardless of the relative social positions. I was taught to do the same in private school in the 80s (though only explicitly to "my elders"), but dropped the habit for the most part, and these days I'm uncomfortable and a bit startled when someone "sirs" me.