On: ProofOfThought: LLM-based reasoning using Z3 theorem proving

[sdenton4]

Indeed - human judges suck on average. And you can prompt an llm judge to look for particular kinds of problems, then throw the ensemble of judges at an output to nitpick. (Essentially, bake in a diversity of biases through a collection of prompts.)