On: Micro-agent: make an AI write code until it passes an unit test

[shakna]

Most LLMs struggle to even do a null check. Could this check for those kinds of glaring security holes?

[jejeyyy77 → shakna]

add a test for it?

[shakna → jejeyyy77]

Kinda hard to right a test that a value that is null-checked, when that value may never actually be returned.

For example, have a C function that reads in a file and returns you a string? The string can be checked that malloc actually succeeded, but how do you check that the file actually opened?

[jejeyyy77 → shakna]

what?

[shakna → jejeyyy77]

Every time you call fopen, you need to do a null check. Every single time. You also need to check that fclose is there to match the call, every time.

Writing a test for that, when it is generally just a call within the function you want to test, isn't really possible. It's not there in the arguments, or the return value, of the function.

How do you check for the right checks in a function expected to do something like this:

int foo() { FILE *fp = fopen("test.in", "r"); if(!fp) { return -1; }

  for(int i = 0; i < NUM; i ++){
   if(matcher(fp, i)){    
    fclose(fp);
    return i;
   }
  }

  fclose(fp);
  return 0;
 }

[jejeyyy77 → shakna]

you write 2 unit tests for your function... one with a test file that exists, and one with a fake file path. Assert not null and null respectively..