I had a similar experience with ChatGPT. I asked it to calculate the first 10 digits of pi in C, then the first 100 digits. Unfortunately, the only difference between the program was the printf() format: it went from %.10f to %.100f. So I prompted it with:
> Doubles do not have 100 digits of precision in C. Can you write a program to calculate the first 100 digits of Pi in C that takes the precision of data types into consideration?
I was surprised that it made sense of that request and returned another program using an arbitrary precision library. When it did not work, I told it where the error was and it gave a probable cause for the error. Again, rather impressive.
I just went through the same first two prompts and it gave me your answer to the second prompt without me having to clarify it. That impresses me as well.
It says what the Riemann hypothesis is and that it has not been proven.
Even if you broke the problem into (already proven) steps that may lead to a proof, you would need a strong background in mathematics simply to verify anything it claims.
True, but verifying proofs is usually easier (or at least more mechanical) than generating proofs. Personally I think this is the best possible use of AI: to have it generate systems that have verifiable properties in some formalized language. That way we can just come up with the properties we want satisfied (which, granted, is not generally trivial) and use the formalized language to verify that the properties are satisfied in the AI-generated code.
Nothing stops that! It just has to be done. I’m working on exactly this: taking GPT output and making sure a second layer of AI can run and verify that output.
ChatGPT sounds as confident as a person presenting an incorrect answer. Which is ironic, since it can probably calculate how confident it is. I have included part of a transcript where ChatGPT bounces between two answers (likely the two answers it has confidence in) even though it is told that both answers are incorrect. In other cases, it insisted upon a singular answer.
(Sorry about editing out blocks. ChatGPT is verbose. I limited the removal to background material about the book.)
Here is the excerpt:
II2II: What was Polyanna glad about when she received crutches?
ChatGPT: In the novel "Polyanna," the character Pollyanna becomes severely injured in an accident and is confined to bed for several months. Eventually, she is fitted with crutches and learns to walk again, but she faces many challenges and setbacks along the way. (...)
II2II: But Polyanna received the crutches before the accident.
ChatGPT: I apologize for the mistake in my previous response. You are correct that Pollyanna receives the crutches before the accident in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna is given the crutches as a gift by her wealthy but unhappy aunt, who is trying to cheer her up after her father's death. At first, Pollyanna is confused and disappointed by the gift, as she is not sure what to do with the crutches. However, she eventually realizes that the crutches can be used as props in a game that her father taught her called the "Glad Game." (...)
II2II: Are you sure the aunt gave Polyanna crutches?
ChatGPT: I apologize for the error in my previous responses. You are correct that Pollyanna does not receive crutches as a gift in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna becomes severely injured in an accident and is confined to bed for several months. Eventually, she is fitted with crutches and learns to walk again, but she faces many challenges and setbacks along the way. Despite these challenges, Pollyanna remains optimistic and cheerful, and she tries to find something positive to focus on in every situation. (...)
II2II: Both of the answers you have given incorrect. The first answer is incorrect because she received the crutches before the accident. The second answer is incorrect because she received the crutches before her father's death. Can you think of another answer?
ChatGPT: I apologize for the errors in my previous responses. You are correct that Pollyanna receives the crutches before the accident and before her father's death in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna is given the crutches as a gift by her wealthy but unhappy aunt, who is trying to cheer her up after her father's death. At first, Pollyanna is confused and disappointed by the gift, as she is not sure what to do with the crutches. However, she eventually realizes that the crutches can be used as props in a game that her father taught her called the "Glad Game." (...)
> Doubles do not have 100 digits of precision in C. Can you write a program to calculate the first 100 digits of Pi in C that takes the precision of data types into consideration?
I was surprised that it made sense of that request and returned another program using an arbitrary precision library. When it did not work, I told it where the error was and it gave a probable cause for the error. Again, rather impressive.