On: Natural Language Autoencoders: Turning Claude's Thoughts into Text

[skybrian]

Training a model is more like evolution. The motivation to "cheat" comes from the evaluations giving it a higher score for "cheating." Change the game and the motivation goes away.

There's no other motivation to be misaligned besides getting higher evals. These goals, plans, subterfuges need to somehow be useful for getting higher evals, or a side effect of them.

[astrange → skybrian]

> The motivation to "cheat" comes from the evaluations giving it a higher score for "cheating."

That's what Goodhart's Law is! All evaluations will eventually cause cheating on them.