Prompt Injection Through Poetry

In a new paper, "Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models," researchers found that turning LLM prompts into poetry resulted in jailbreaking the models: Abstract : We present evidence that adversarial poetry functions as a universal single-turn...

Jailbreaking in the Haystack

Recent advances in long-context language models LMs have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety implications of these extended contexts remain unclear. To bridge this gap, we introduce NINJA short for...

The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning

Large Language Models LLMs have advanced rapidly and now encode extensive world knowledge. Despite safety fine-tuning, however, they remain susceptible to adversarial prompts that elicit harmful content. Existing jailbreak techniques fall into two categories: white-box methods e.g., gradient-base...

Active Honeypot Guardrail System: Probing and Confirming Multi-Turn LLM Jailbreaks

Large language models LLMs are increasingly vulnerable to multi-turn jailbreak attacks, where adversaries iteratively elicit harmful behaviors that bypass single-turn safety filters. Existing defenses predominantly rely on passive rejection, which either fails against adaptive attackers or overly...

NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models

In deployment and application, large language models LLMs typically undergo safety alignment to prevent illegal and unethical outputs. However, the continuous advancement of jailbreak attack techniques, designed to bypass safety mechanisms with adversarial prompts, has placed increasing pressure ...

PurpCode: Reasoning for Safer Code Generation

We introduce PurpCode, the first post-training recipe for training safe code reasoning models towards generating secure code and defending against malicious cyberactivities. PurpCode trains a reasoning model in two stages: i Rule Learning, which explicitly teaches the model to reference cybersafe...

Evaluating the Critical Risks of Amazon'S Nova Premier under the Frontier Model Safety Framework

Nova Premier is Amazon's most capable multimodal foundation model and teacher for model distillation. It processes text, images, and video with a one-million-token context window, enabling analysis of large codebases, 400-page documents, and 90-minute videos in a single prompt. We present the fir...

Watermarking Degrades Alignment in Language Models: Analysis and Mitigation

Watermarking techniques for large language models LLMs can significantly impact output quality, yet their effects on truthfulness, safety, and helpfulness remain critically underexamined. This paper presents a systematic analysis of how two popular watermarking approaches-Gumbel and KGW-affect...

GraphAttack: Exploiting Representational Blindspots in LLM Safety Mechanisms

Large Language Models LLMs have been equipped with safety mechanisms to prevent harmful outputs, but these guardrails can often be bypassed through "jailbreak" prompts. This paper introduces a novel graph-based approach to systematically generate jailbreak prompts through semantic transformations...

ChatGPT Security: OpenAI's Bug Bounty Program Offers Up to $20,000 Prizes

OpenAI, the company behind the massively popular ChatGPT AI chatbot, has launched a bug bounty program in an attempt to ensure its systems are "safe and secure." To that end, it has partnered with the crowdsourced security platform Bugcrowd for independent researchers to report vulnerabilities...