Speaking mathematically, you are right. However, linguistically I disagree. Consider: Someone tells you that "all of their kids are doing great in school". Turns out they have no kids. They obviously were trying to deceive you, and make you think they do have kids - in fact, since plural, more than one kid. Hence, it is effectively a lie.
So if the liar speaks of "all my hats" while having none, that is deceptive. I would consider it a lie.
A wife asks her programmer husband” on your way home, can you swing by the store and buy one carton of milk, and if they have eggs, get six?”
It’s funnier and more relatable to programming if he comes home empty handed, crashes the car into the garage door, and says, with perfect alacrity, “six what?”
More generally, the puzzle is kind of stupid -- not as a puzzle, but as a representation of life -- because speakers do not speak according to the rules of mathematical logic. That doesn't make them liars, it just means they don't agree about the ground rules.
What does it even mean to be right mathematically here? If I invent a mathematical structure where I define elements 1 and 2, an operation + and a relation = that posits that 1+2=2, I can say that mathematically one apple plus two apples equals two apples. Would I be mathematically right or would I be applying a wrong/not-even-wrong/linguistically deceiving /incoherent model to the real world?
For what it's worth (to whomever was upset by this): this is not apologetics—there's nothing here in my comment to give anyone cover for being dishonest. It is sufficient for something to be dishonest in order for to it to be deserving of all the judgement that people have for liars. It is the dishonesty that is bad, whether it takes form of a lie or not.
But conflating dishonesty with lying is harmful, because once you do that, you give ammunition to people who employ dishonesty in instances that don't involve lying, because if everyone is taking it as a given that dishonesty and lying are the same, and they can show that they weren't lying, then they can argue they weren't being dishonest. But that's wrong since dishonesty and lying are not synonymous—people can still be dishonest without lying—and, again, it is the dishonesty that is bad.
So if the liar speaks of "all my hats" while having none, that is deceptive. I would consider it a lie.