When Scanners Lie: Evaluator Instability in LLM Red-Teaming

Automated LLM vulnerability scanners are increasingly used to assess security risks by measuring different attack type success rates ASR. Yet the validity of these measurements hinges on an often-overlooked component: the evaluator who determines whether an attack has succeeded. In this study, we...