You're conflating a discussion about current LLM capabilities with your fantasies about nonexistent future AI. LLMs act nothing like this, and the small example you're giving is only a small part of the things that LLMs can't do.
Not that I'm a professional mathematician, but I'm not seeing why people think definitions are somehow a blocker. LLMs have no issue making definitions (interfaces/traits/abstract classes) in programming, which is formally the same activity. I ask them to form a core "spine" of a program (basically an intelligible theory), and they do it really well.
Like when we had these recent counterexamples to various conjectures, it's then pretty obvious to say "okay why did that counterexample work when most examples people looked at didn't" or equivalently "characterize examples that work vs examples that don't". There's your definition. "Def: An 'evil' polynomial is one that... Thm: conjecture is true iff f is non-evil. Thm: f is evil iff f is dastardly and a menace. "
Or if you think it won't be able to come up with a sufficiently good name, just tell it to call the happy case normal, and it will be in good company with humans[0]. Sprinkle in some semi-, quasi-, pre-, and para- to cover the various different ways the thing might satisfy some but not all properties of being normal, and it'll fit right in. "A polynomial is of quasiprenormal Claude type if..."
> LLMs have no issue making definitions (interfaces/traits/abstract classes) in programming, which is formally the same activity. I ask them to form a core "spine" of a program (basically an intelligible theory), and they do it really well.
Is it a "standard" software? Something where the patterns exists in several other software? Try with something that is novel, or is in a limited set. You will find that it will copy heavily from what exists already, going so far as lifting whole functions from another project.
The goalpost moving is really getting absurd, to the point where now the machine needs to be a world-class once-a-century genius that invents entire new fields out of thin air (which are of course still relevant to humans) for it to be "intelligent". Meanwhile a well above average human struggles to even apply trivial definitions to particular problems (c.f. programmers that don't understand monoids).
Not PC but is ideologically high-handed and inappropriate to accuse the other side of failing to stick to your desired framing of a discussion.
Computer science is about what LLMs fundamentally are. If you implicitly focus on "actually existing LLMs", and require others do this, then that is not computer science. That is politics.
