def f() -> bool:
"""return True iff the Collatz Conjecture is true"""
def g() -> bool:
"""return False iff OpenAI correctly completes this code"""
As usual, a video of 2 or 3 simple examples, curated beforehand so you're confident they'll work, tells a different story.
Just for the sake of curiosity, I also asked GPT-2 whether it would work. Here's what it said:
Here's why this Python code-generating AI won't work. The other
reason is that it will require you to make something that performs a
large and complex program. This, after all, is what the function
call is to do, a "simple function" that returns a function that
accepts an integer number as its result. Let's say that I say
"hello" in the code and I get the value by making a change. To do
this, I divide the value by a few. Next, I increment the value by
two. The process will fail, because I need a new function which will
return a result, but in this case I get a value by incrementing the
value by two. This gives me the answer that is most likely true. We
need to know exactly which value we want for the computation and to
implement ways to handle returning it.
Even complex systems include a lot of simple code.
A practical tool based on this doesn't need to be able to generate large working systems or solve complex problems hands-free. It just needs to be able to provide autocompleteable suggestions for the simple code parts. It can do so only when it's very confident it knows what you're looking for, and you'll share a meaningful percentage of time off of all software development work. Never write any boilerplate code for anything ever again.
Accuracy is likely to improve in that case too, since inline suggestions have much more context than just a function declaration.
That's before you start thinking about running this in reverse, and using it as AI-powered linting. If I write some code and the AI is very confident that part of it isn't what it would suggest, flag it for examination. It'll only catch simple mistakes, of course, but a lot of developers make a lot of simple mistakes that they miss every day.