One thing I'll tell you is that in my experience, every culture has its own way of expressing annoyance. When those ways clash, it can make people seem way more hostile than they mean.
As an example, if I start a sentence with "Sorry, but..." I'm pretty annoyed, but in some parts of the world, that would be endearing. In other parts, it would be so polite as to be annoying.
One thing I'll tell you is that my country isn't particularly patient with people who don't try to learn English, or with people whose English is poor. We have a reputation for being very polite and welcoming, but I think that's more a function of people just not picking up on cues that we're annoyed.