Yes, it's presumably a problem, but you also can't know why somebody is downvoting. They might believe a comment is dishonest or distracting or otherwise low-quality or non-curious, or breaks one of the subjective site rules.
I try not to make assumptions when I get downvoted. I think: Maybe I was a little sarcastic or aggressive or I implied something weaselly; or maybe the downvoter just disagreed with me. I don't assume either way, and I don't guess about moral high ground or whatever. All my comment's points tell me is what delta of people clicked the down arrow, not what was in their heads individually.
Left you an upvote because I'm interested in responses to this.