Allowing someone to do something means there is estoppel (my explicit action to let you do X is considered in law to prohibit me later going after you for doing X even if you'd otherwise have no right to do it) but doesn't magically abolish my right to forbid others.
Claims require that you actively use the mark, if you last sold Rocqua brand hats in 1987, your claim against me for using that mark in 2019 won't work, but they don't require you go around suing or even threatening to sue anybody. That's just lawyers getting themselves a few extra bucks off the unenlightened, mixed with people who want an excuse for behaving badly.
Not only is there a risk of estoppel (probably not if they can't show anything explicit or intentional by you), even other competitors can point to your willful inaction in support of their argument to a judge that your trademark has become generic.
As noted above, I'm not a lawyer or giving specific legal advice here, but I have worked with lawyers in dealing with trademarks.