OK, so if I understand correctly, I formulate a proposition in natural language, then an LLM creates 250k pages of proof based on it, then it takes those 250k pages/lines of proof and creates 15 million lines of code to prove those 250k pages. Is that correct? If so, how can I be sure that my proposition is correctly translated into those 250,000 pages? Isn't it possible that the LLM can reward hack here in the same way I've seen it do in some cases where it struggles to pass the test and get its reward? For example, it tries to change assumptions without telling me because something doesn't work or it removes something or changes the test etc.
What assumptions could it change? The model should not be able to alter the definition of the RH, and if you want to prove to yourself that that definition is accurate, you just have to read through it and build a mental model of the Lean code: https://github.com/bhgomes/lean-riemann-hypothesis
Then if it constructs a valid proof in Lean, how could it be reward hacking? I suppose it could have found a Lean compiler bug, or a mistake in the formalization of the hypothesis. I would expect that those problems would be found and fixed reasonably quickly, and we would need to see if the proof could be fixed. Other than that, I suppose it could find a contradiction in ZFC, but that would be an even more incredible result than proving RH.
I don't really see the possibility for reward hacking with the stipulation that the proof is ultimately verified by compiling the Lean program.
>> If so, how can I be sure that my proposition is correctly translated into those 250,000 pages?
It's not the proposition (i.e. the theorem) you need to worry about but the proof. I'm sorry to say but the comments on here (and everywhere on the internets) that assure each other that "if it's veified by Lean, then it's true" misunderstand what they're even trying to claim.
Here's a good article on the subject that was recently posted on HN:
>> This post is an attempt to explain in simple terms how to interpret claims of the form “this theorem is correct because it was checked in Lean”. The summary is: there is an important signal that can be extracted from Lean code, but if this code is AI-generated then you need to do some tests first. We explain the tests, and note that in many cases (such as the OpenAI theorem dump) these tests have not yet been systematically done.
> For example, it tries to change assumptions without telling me because something doesn't work or it removes something or changes the test etc.
You have in Lean a headline statement of "given X, then Y is true". Where X and Y and precise statements in code. So the mathematical assumptions are upfront.
If you trust the lean compiler and are very careful to vet that the headline statement is what you expect, then you know the theorem is true.
How well the human language mechanics of the proof corresponds to what the actual lean mechanics of the proof is very much up on the air.
I see, so there is one step less, then how I can know if that 250k pages describe my proposition correctly and not something else? Lean code was written by LLM, correct?
In formal proof verification the only thing you need to verify is the problem statement expressed in lean, you don't need to check the lean code that is the proof. If the lean code that is the proof compiles, its a verification of the proof.
In theorem proving environments like Lean, you are equipped with a standard set of axioms and statements (eg set theory). Most people, as far as I am aware, agree with Set Theory. From these basic fundamentals, you can build up all of mathematics, just like a normal mathematician can. The trick is that in systems like Lean, proofs are properly type checked, like a programming language; and just like a programming language, those types help stop bugs in the proofs, but the types here are far richer, able to encode all of mathematics, and far more strict. To prove a mathematical statement, you must write a proof that is accepted by the type checker. Beyond that, the core "language" itself is generally specified to be "sound" -- that is, there is a very small subset of Lean, a tiny "kernel language", that is shown to be incapable of proving False, and thus any proof which is well typed in said language (ie it typechecks) is also sound.
In other words the system is designed, by construction, to only rely on a very small "trusted" set of axioms, code, and mathematical ideas. This is a very engineering-oriented design principle! Actually, a pretty significant amount of the theoretical research in theorem proving over the past 2 decades has been in "automating proofs", ie tools and methods of making those proofs as easy as possible to write. Entire book chapters have been written about how to replace a 20 line inductive proof in Rocq with a single line "function call" that can automatically discharge the proof for you, because the idea is that this "trusted kernel", the tiny and sound mathematical language, will reject invalid proofs by design. So "who cares" what the proof is, in a sense? In many ways simply writing the theorems and whatnot down requires you to understand the problem well enough already. So in a way LLMs are just taking this idea to the absolute limit.
So all this means is that the difficulty really is in specifying the theorems. Which was kind of the whole problem all along! Except now you have a computer that can check your actual proof. You can write a theorem in Lean that isn't the theorem you want, and then write a type checked proof of it, but you still need the theorem to be accurate.
Unless Claude is either 1) finding bugs in the Lean kernel to exploit them, 2) using 'sorry' (a very convenient tool in Lean used as a placeholder for proofs, but it can literally allow you to prove False), then it mostly comes down to: are all the theorems stated correctly? This is trickier than it sounds, of course, so some skepticism is certainly warranted in my opinion!
There are also other various engineering principles applied to help try to reduce bugs and ensure correctness like any other engineering project, and there are various other "things" regarding dependently typed programming language design and the impact of those design decisions on proofs (and why they are notoriously hard to read and write), but they aren't germane to my point here.