1. The intended audience, whether an individual or group.
2. Anyone who could generically presume you somehow meant them or were commenting on their life.
3. Actual people you personally know that you actually are speaking about or that might legitimately assume you meant them when you didn't.
Group 3 is probably the biggest source of trouble for most people. It's the one I make the most effort to account for.
My brother-in-law is (or was) a programmer. Though his hourly rate is certainly something I envy, he has a history of working part-time and intermittently and is more talented at spending money than at making it.
That strongly shapes my failure to live in awe of programmers and my failure to presume them to all be wealthy and powerful types. But I believe this is the first time I have said such on HN because it's challenging, at best, and potentially impossible to express that without sounding like I am putting him down. So I just never said anything about it. Most people didn't really need that information anyway, so not saying something potentially offensive to my sister, her husband and other relatives was the easy solution.
Basically, before you tell that cutesie anecdote about your own life, contemplate what it says about people connected to you and how they might feel if they heard you telling that story. Does it cast them in a bad light? Could it be construed as such, even if that wasn't your intention? Is there some means to tweak it to make it less problematic?
The other thing I worry a lot about are acquaintances that will think I meant them when I didn't. I try to not make comments that could be taken to mean I am talking about them when I'm actually not. This is often a matter of tweaking it slightly. It might be as simple as saying "a curly haired person I know" instead of "a blonde person I know" to make sure a blonde acquaintance with straight hair doesn't assume I mean them.
Group 2 is best addressed by finding a sympathetic framing and going ahead and giving some provisos. Don't assume that folks will just know that, of course, you would make allowances for X.
Your friends and relatives may know that, but other people won't. It helps to view it as simply an artifact of clear communication.
The mistake most people make is only thinking about the intended audience and stopping there. You do need to think about that. But you should also think about anyone you are "talking about" or could be construed as talking about.
Edit: To be perfectly clear, I'm not putting my brother-in-law down. His work history is due in part to supporting my sister's career, a thing she doesn't appreciate enough.