Indeed, the Microsoft Copilot eco-system might be a bit more sophisticated these days.
It so just happens than people around me, including myself, don't use the copilot, we "left" for the next big thing when Cursor was release, and copilot was still a glorified auto-complete.
From your feedback it seems like they became quite good?
I actually though about it multiple times over at this point.
You're right, this deserves more attention, and is a valid problem going forward with this app. And I had this problem when just started building, it either generated XSS tests for any user input validation method (even if it used other validators) or just 1 single test case.
For now I attempt to strictly limit the amount of tests for LLM to generate.
This is achieved with "Planner" that plans the tests for each function before any generation happens, that agent is instructed to generate a plan that follows the criteria:
- testCases.category MUST be one of "happy_path" | "edge_case" | "error_handling" | "boundary".
And it is asked to generate 2-3 tests for each category. While this may result in the unnecessary tests, it at least tries to limit the amount of them.
Going forward I believe the best approach would be to tune and tweak the requirements based on the language/framework it detects.
Hey, thanks for the feedback, i will make sure to make it more visible/less confusing. So the model is actually quite simple.
1 credit - 1 file up to 15 functions. <-- only this tier is available in alpha, due to current limitations in the implementation, i tried generating on bigger files and it took quite a long time, so i am in the workings on solving this issue before enabling larger files support.
2 credits - 1 file up to 30 functions.
3 credits - 1 file 30-35 functions.
P.s if generated tests have <70% pass rate (at which point probably something went horribly wrong, your credits are refunded)
So from my experience with the LLMs if you ask them directly "is this a bug or a feature" they might start hallucinating and assume stuff that isn't there.
I found in a few research/blog posts that if you ask the LLM to categorize (basically label) and provide score in which category this issue belongs it performs very very well.
So that's exactly what this tool does, when it sees the failing test it formulates the prompt in a following way:
## SOURCE CODE UNDER TEST:
## FAILED TEST CODE:
## PYTEST FAILURE FOR THIS TEST:
## PARSED FAILURE INFO:
## YOUR TASK:
Perform a deep "Step-by-Step" analysis to determine if this failure is:
1. *hallucination*: The test expects behavior, parameters, or side effects that do NOT exist in the source code.
2. *source_bug*: The test is logically correct based on the requirements/signature, but the source code has a bug (e.g., missing await, wrong logic, typo).
3. *mock_issue*: The test is correct but the technical implementation of mocks (especially AsyncMock) is problematic.
4. *test_design_issue*: The test is too brittle, over-mocked, or has poor assertions.
Then it also assigns the "confidence" score to it's answer, based on that either full regeneration of the tests proceeds, commenting on the bug in the test, fixing mocks or full test redesign (if it's to brittle)
While this is not 100% bullet proof, i found this to be quite effective way - basically using LLM for the categorization.
To clarify, each failing test triggers "review" agent, to determine "why" the test fails, and again, it can be improved with better heuristics probably, more in depth static analysis than the source code, but it is how it works in the current version.
i wonder if always having a design doc of some substance discussing the intended behavior of the whole app would help reduce instances of hallucination. The human developer should create it and let it be accessed by the AI
It so just happens than people around me, including myself, don't use the copilot, we "left" for the next big thing when Cursor was release, and copilot was still a glorified auto-complete.
From your feedback it seems like they became quite good?
reply