FWIW, I believe that it is more "you can lose it if you lose control of it"; just because you haven't enforced it is not a big deal if the people you haven't enforced it against are sufficiently small that they wouldn't have been noticed in a wide enough audience to undermine your trademark.
(In essence, the argument is not a technical one: it is only interesting in that it changes the dynamics of "could someone have been confused"; if people see the word "Facebook" as a moniker on products all the time that are not made by Facebook, then clearly they are not going to be confused by any individual one.)
I sort of agree: You don't automatically lose anything. But once you sue to enforce your trademark, the court looks at the body of evidence - and may determine that the trademark has been diluted and has been lost. There is no AFAIK predefined threshold (or way to measure, for that matter) the dilution of trademark - a different judge/jury may reach different conclusions.
And basically, that's why most trademark lawyers will act on even the smallest unauthorized trademark use - they do not want to leave it to chance.