You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.
Based on the SynthID-Text paper https://www.nature.com/articles/s41586-024-08025-4 I agree that the LLM's learned distribution isn't modified, but I don't think it's correct to say that the sampling process is not modified. Also I just read the paper today so I could be misinterpreting things.
As described in the paper, you're right that it doesn't affect the main sampling technique, but what they do is they sample the distribution for 2^m samples, and then use Tournament sampling to choose the tokens among those 2^m samples, and the watermark key changes the scoring of the tournament options, using the watermark key as an input to the random generator that generates the scoring functions.
Then, to calculate the watermark, they take the text, and compute the mean g-values of the text, and a higher score means that it's more likely that it was sampled using the provided selection of tournament watermarking functions.
let's say you had some top P words: mango, banana, pineapple, guava, and you sampled 8 times, and got each one twice in the following order:
without tournament sampling, you'd truly see any of those come through. But in tournament sampling, you take those 8 options, create m scoring functions based on the pseudorandom generator, and score the 'tournament' by sampling the biased distribution you create from the g values. That does change the sampling from based purely on the LLM and entropy, but i mean, if the watermark key is also generated from some entropy, it's probably representative of the original sampling options as expected?
this is a very fascinating topic! I do still stand by my point that anthropic is the only one who can tell if something is watermarked or not and feeling icky, but the paper has mostly quelled my concern on impacting the intelligence part.
I think you're correct. It does alter the distribution for each output token, implicitly giving each candidate token a different probability. Maybe that's fine, but it's not as magical as Anthropic [and a lot of commenters here] are making it out to be.
Finally convinced myself that non-distorting watermarking is real, a la Anthropic / Google SynthID. It's not just spin / marketing. This really is a "Monty Hall" like problem. (h/t @random_walker for that analogy.)
Sharing here in case it helps anyone else.
On one hand, watermarking is "obviously" damaging to the output. The LLM does all this work to compute token probabilities... then you essentially perturb the probabilities? Of course that's bad! Every single token is using perturbed probabilities!
On the other hand, you obviously can freely inject a layer of randomness into sampling. Take any categorical distribution. For each sample, perturb the probabilities, then draw the sample. Done correctly, over many samples the results will match the original distribution.
It still kinda feels like magic, but it's not surprising that you can take that core trick and shape it into a non-distortionary watermarking scheme.
(i.e. it gives you sequences that were just as likely to be generated by the original model as any other sequence)
