Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

Jailbreak attacks against large language models LLMs are evaluated using attack success rate ASR, which depends on correctly identifying responses that fulfill prohibited requests. Existing evaluators can mistake refusals or topically related responses for successful attacks. We show that changin...