> But it’s virtually useless for editing or changing code where it will often produce code that doesn’t compile, has bugs and/or doesn’t solve the requirements.
That is the exact problem I am trying to solve: modifying code with LLMs really sucks most of the time. I am trying a solution with Abstract Syntax Trees: I have the LLM write the code that will write the code you need. That is, modify the source tree rather than the text representation.
I do have it working for some cases quite well, but there are lots of pitfalls with this approach too. It does take a lot of context and the LLMs aren't really that well-versed at writing specifically esprima code for example. BeautifulSoup does work better, I guess because more people use it and there's more data in the training set.
I'm adding one language at a time, currently have HTML, CSS, Javascript and Python all kind-of working. It's pretty neat but I'm not sure how well it scales yet to larger projects and more difficult requirements/implementations.
That is the exact problem I am trying to solve: modifying code with LLMs really sucks most of the time. I am trying a solution with Abstract Syntax Trees: I have the LLM write the code that will write the code you need. That is, modify the source tree rather than the text representation.
I wrote about my approach here: https://codeplusequalsai.com/static/blog/prompting_llms_to_m...
I do have it working for some cases quite well, but there are lots of pitfalls with this approach too. It does take a lot of context and the LLMs aren't really that well-versed at writing specifically esprima code for example. BeautifulSoup does work better, I guess because more people use it and there's more data in the training set.
I'm adding one language at a time, currently have HTML, CSS, Javascript and Python all kind-of working. It's pretty neat but I'm not sure how well it scales yet to larger projects and more difficult requirements/implementations.