Sockpuppetting: Jailbreaking LLMs without Optimization through Output Prefix Injection

As open-weight large language models LLMs increase in capabilities, safeguarding them against malicious prompts and understanding possible attack vectors becomes ever more important. While automated jailbreaking methods like GCG Zou et al., 2023 remain effective, they often require substantial...