I asked Chat-GPT to prove that the set of all integers is uncountable (it isn't). What's interesting is that Chat-GPT not only spat out the classic diagonalization proof, but rephrased around integers where it doesn't work, it ended with "This may seem counterintuitive, because we know that the integers are countable, but the proof clearly shows that they are uncountable."
Not only will Chat-GPT mess up math on its own, you can ask it to mess up math and rather than refuse, it cheerfully does it.
Not only will Chat-GPT mess up math on its own, you can ask it to mess up math and rather than refuse, it cheerfully does it.