You are the very first person I encountered who claim LLM can fix technical debt. Usually I seem comments and articles saying LLM can only produce it. And I can agree with those articles and comments
Here's the result of a prompt that specifically looks for correctness and simplifications to make "by construction" before I added "Design bar: ensure correctness by construction rather than by convention" to AGENTS.md:
I did. Just about all of the findings were re-verified to be good and then implemented by a rigorous pipeline of clankers, and the project was improved.
If it weren't for me insisting that I manually sign off on the solution of every finding, then it would have been fully automated too.
There is a lot of noise out there from anti-AI types who never seriously tried to clean up messes with AI. If you just ask for new features and never look at the code, coding agents will make a mess. If you often ask questions about how to clean up messy code, coding agents are great help for that, too. But you have to ask.
Agents (and humans) don't know what cleanups are safe or a good idea. They can get stuck in loops when different goals (conciseness, performance, etc.) clash. Their "improvements" can break things unintentionally. Targeted improvements can be good. Giant lists of "fix everything" are bad and will break things.
Tech debt comes with insufficient tests, so you won't know what you've broken until too late in many cases.
Yeah that's why step 0 is to tell the LLM to write a test suite for the thing you want to rewrite. It'll do that, but it does require adult supervision. Once you have that test suite, you can send the LLM into an inner dev loop. Like human programmers, LLMs are great when they have a fast inner loop.
The fact that LLMs exist doesn't mean you can just stop thinking. It mean the things you have to think about will be different. You have to treat them as savants with absolutely no ambition.
There’s a trap here though: the LLMs if you let them will spend all their time maintaining that test suite, they can become quite rigid in the way they design them. So it’ll write some code, break a test, which breaks a contract so then it updates the test and the contract but updating the test also broke another contract… soon enough you’re spending all your tokens fixing contracts instead of writing code.
In a sense the LLM having a fast inner loop is its blessing and curse. A blessing because it gets feedback quickly, but a curse because it becomes naval gazing and cannot see the forest for the trees.
At least this is my experience with Sol maby other models behave differently.
I’ll be the second then. I’ve found it’s very good at creating debt but equally good at resolving it. The thing is the default mode is creating debt so you have to intentionally pay it down.
Refactors that used to take me a month now take a week, it’s very handy. You can instruct them to move functions around, change interfaces, add or remove abstractions, remove redundant authorities, untangle spaghetti, rename identifiers across a codebase, and it will return very good results.
Yes like any tool, LLMs can be used poorly. But they do enable large-scale rewrites or poor-quality code because they are capable of understanding it and enumerating its behaviors in a matter of minutes or hours. It's the kind of understanding that would take a human months or years to develop.
Add to that their ability to basically set up what is essentially a perfect testing environment when asked, and you've got a feedback loop that lets you just step back while the agent cranks out a new implementation in a memory-safe language with a full test suite and bug-for-bug compatibility. I'm not kidding or exaggerating. This stuff is possible now, people just need to look past their anxieties about being replaced.
