False Sense of Security: Why Probing-Based Malicious Input Detection Fails to Generalize

Large Language Models LLMs can comply with harmful instructions, raising serious safety concerns despite their impressive capabilities. Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in LLMs' internal representations, and researchers ha...