I don't usually point out mistakes in non-native speakers' writing because I know most people will perceive it the way that you do. However, I always wish that people would correct me when I make a mistake in a foreign language. So part of me always wants to point things out to help people like me.
Like most complex interactions in life, it really comes down to just feeling when the time's more appropriate. Sometimes people explicitly ask you to correct what they just said ("was that right?"), sometimes they halt in the middle of a sentence and look at you expectantly so you just finish/reword it for them, etc. But generally speaking you shouldn't try to correct people too much unless it really impedes proper understanding, it can really undermine people's confidence in their speaking ability even when they are otherwise fluent. And when you do correct them, only do so with respect to grammar or improper choice of words; correcting pronunciation can be a minefield given how it can change widely within a native population, what with accents and suchlike. (Some people, especially in such a centralized language featuring very normative institutions, will argue that they don't have an accent and everyone else does, but they're wrong.) It should only be corrected when it would otherwise lead to misunderstandings.