Out of curiosity, I tried to use freely available LLM to generate simple Python tests. I provided the code and specified exactly what requirements I want to be tested. What I found out is that initially it generates repetitive, non-DRY code so I have to write propmts for improvement like "use parametrization for these two tests" or "move this copy-paste code into a function". And it turns out that it is faster to take initial version of code and fix it yourself rather than type those prompts and explain what you want to be done. And worse, the model I was using doesn't even learn anything and will make the same mistakes the next time.
But, this was a generic LLM, not a coding assistant. I wonder if they are different and if they remember what you were unhappy with the last time.
Also LLMs seem to be good with languages like Python, and really bad with C and Rust, especially when asked to do something with pointers, ownership, optimization etc.
But, this was a generic LLM, not a coding assistant. I wonder if they are different and if they remember what you were unhappy with the last time.
Also LLMs seem to be good with languages like Python, and really bad with C and Rust, especially when asked to do something with pointers, ownership, optimization etc.