This is an interesting experiment. Curious why you used LLM-as-a-Judge (gpt-6-astra). Did we have a ground truth of actual RCA done by a human to compare against?
Yes! All faults in SREGym-lite (and the parent benchmark SREGym) have their ground truths reviewed by us to ensure that they are accurate to the incident. In addition, we also conducted an analysis to ensure the LLM judge agrees with the human annotators too, which can be found on page 6 here: https://arxiv.org/abs/2605.07161.
