It really does depend on office culture. Some people might be more reasonable if you communicate directly. I don't think it would go down well everywhere if you state "Hey! No talking to me if Mr. Smurf is on the table!"... Mr Smurf might have an unfortunate accident sooner rather than later.
