On: The Tragedy of Google Books (2017)

[ASalazarMX]

You punish it if parts of the answer can be found in its training data, and reward it otherwise.

[amelius → ASalazarMX]

But the whole point of the training is that you reward it if it correctly reproduces the next token.