"You" is pretty rude in America (or at least the North East).
To add a little more context, I work in emergency services (fire/EMS), so if I'm addressing someone, it's typically to give them instruction to keep them safe ("Sir, I need you to ______"). I also use 'sir' before I know someone's name ("Excuse me sir, could I get your name?").
The word can certainly have different connotations depending on the relationship. A service employee would generally refer to a customer/employer as "sir," and I suppose there's an air of subservience there.
On reflection, I suppose you could call a firefighter a "service employee," and civilians on an emergency scene are in many ways "customers," so my use of the term might just be an extension of that relationship...