Having said this, I personally think that it is rude to talk in a language in work settings that someone else does not understand even though you may not be talking to/about them.
However, many times, the people doing this do not realize it is rude (again due to cultural differences). They will think "what the heck, I am talking in my mother tongue to someone else, why does he/she care". They do not realize that it might be making the other person uncomfortable.
P.S: I am an Indian who moved to the US many years ago.