MAI-2024-0053

Large Language Models LLMs that utilize reinforcement learning from human feedback RLHF are susceptible to jailbreaking attacks due to issues with reward misspecification. The reward function employed during the alignment process inadequately assesses the quality of responses, especially when fac...

MAI-2024-0051

Large Language Models LLMs that utilize reinforcement learning from human feedback RLHF are susceptible to jailbreaking attacks due to reward function misspecification. The alignment process involves a reward function intended to evaluate the quality of responses, yet it often fails to accurately...

MAI-2023-0003

A vulnerability has been identified within the fine-tuning API of GPT-4, which permits adversaries to bypass the Reinforcement Learning from Human Feedback RLHF safety mechanisms. By employing a relatively small set of meticulously crafted prompt-response pairs, attackers can fine-tune the model ...

BadReward: Clean-Label Poisoning of Reward Models in Text-To-Image RLHF

Reinforcement Learning from Human Feedback RLHF is crucial for aligning text-to-image T2I models with human preferences. However, RLHF's feedback mechanism also opens new pathways for adversaries. This paper demonstrates the feasibility of hijacking T2I models by poisoning a small fraction of...

MAI-2024-0017

Large Language Models LLMs utilizing Reinforcement Learning from Human Feedback RLHF for safety alignment are susceptible to a sophisticated "alignment-based" jailbreak attack. This attack employs a best-of-N sampling strategy in conjunction with an adversarial LLM to efficiently craft prompts th...