I don't think they're claiming precision loss is a bad thing, but instead is just pointing out that BF16, in exchange for more "dynamic range" (eight bit exponent instead of five), you lose some precision. After all, BF16 is just F32, but with a truncated significand.
Yes, that much I do understand. I'm just confused why that is important when predicting the next token? What about the calculation requires that you keep track of really big numbers (and don't overflow) and requires that you have very precise numbers (and don't truncate the precision)?
Because AI is an alchemical art involving heuristically selecting arcane “activation functions” without critically thinking about precision distribution or symmetry, so they rely on band-aiding over it via altering number formats rather than properly matching the function to the appropriate numeric domain.
