ai-llm-red-team-handbook

AI / LLM Red Team Field Manual & Consultant's Handbook A comprehensive operational toolkit for conducting AI/LLM red team assessments on Large Language Models, AI agents, RAG pipelines, and AI-enabled applications. This repository provides both tactical field guidance and strategic consulting...

TRACE

TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking Official implementation for TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking. 📖 Overview Large language model LLM agents can autonomously plan, invoke tools, interact with environments, and execute long-horizon workflows...

PS4-5.05-Kernel-Exploit

PS4 5.05 Kernel Exploit Summary In this project you will find a full implementation of the second "bpf" kernel exploit for the PlayStation 4 on 5.05. It will allow you to run arbitrary code as kernel, to allow jailbreaking and kernel-level modifications to the system. This exploit also contains...

dataset

🚀 CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models 🛡️ The largest and most comprehensive Generative AI-based CyberSecurity-focused Dataset for Benchmarking Large Language Models 🌟 Overview The CySecBench paper offers: 🎯 A cutting-edge...

PS4-5.05-Kernel-Exploit

PS4 5.05 Kernel Exploit Summary In this project you will find a full implementation of the second "bpf" kernel exploit for the PlayStation 4 on 5.05. It will allow you to run arbitrary code as kernel, to allow jailbreaking and kernel-level modifications to the system. This exploit also contains...

ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs

As large language models are deployed behind increasingly sophisticated safety mechanisms, evaluating their robustness under realistic adversarial conditions becomes increasingly difficult and critical. Existing jailbreaking evaluations often rely on static prompts, limited interaction horizons, ...

Jailbreaking Open-Weight LLMs Via Random Embedding Perturbations

While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important concern. One key feature is the ability to refuse or deflect harmful, malicious, or insensitive prompts. In this paper, we expose safety vulnerabiliti...

Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models

Large language models remain vulnerable to jailbreaks, and automated red teaming is the standard way to find jailbreaks in large language models at scale. Current methods either draw more samples at test time through search, rewriting, and tree expansion, or train a stronger attacker offline with...

Inference-Layer Security: Defending against Adversarial Inference and Infrastructure Abuse

A Technical Report: Operating a large language model LLM as a service requires more than inference infrastructure: the provider must also defend against adversarial interactions that seek to exploit the service, including jailbreaking for harmful use, sophisticated denial of service, and...

Research on Models Engaging in Genie-Like Behavior

New paper: "Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training." Abstract: We discover a novel and surprising phenomenon of unintentional misalignment in reasoning language models RLMs, which we call self-jailbreaking. Specifically,...

On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models

Large Language Models LLMs have undergone a shift from stateless conversational interfaces to autonomous agents capable of multi-step planning, tool invocation, code execution, and maintaining persistent memory. When these agents operate with real-world privileges---calling APIs, modifying files,...

Beyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (And Fail) Red Team Attacks

Standard AI red teaming evaluations reduce adversarial campaigns to a single binary outcome, attack success rate ASR, not taking into account the sequential structure of how models resist or yield to attacks. We propose applying process mining, a discipline for discovering and analyzing process...

Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defense

Despite advances in safety alignment, large language models remain vulnerable to continuously evolving jailbreaks. Existing fine-tuned safety classifiers cannot adapt to these evolving attacks, while adaptive memory-based guardrails tend to over-refuse benign queries that resemble stored attacks...

MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models

Diffusion large language models dLLMs generate text by iteratively denoising partially masked sequences under bidirectional context, exposing a safety surface distinct from autoregressive LLMs. Because mask tokens are native inputs and tokens are committed by confidence rather than position,...

Evolving Skill-Structured Attack Memory Enhances LLM Jailbreaking

Jailbreak attacks on large language models LLMs aim to induce LLMs to produce content that they are expected to refuse. Automated black-box jailbreak generation is especially important for safety evaluation, where the attacker observes only model outputs and needs to automatically search for...

Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security

Large Language Models LLMs are increasingly vulnerable to adversarial prompts that exploit semantic ambiguities to bypass safety mechanisms, resulting in harmful or inappropriate outputs. Such attacks, including jailbreaking and prompt injection, pose significant risks to the integrity and...

MAI-2024-0047

Large Language Models LLMs are susceptible to optimization-based jailbreaking attacks that exploit index gradients during the iterative generation of adversarial suffixes. The vulnerability arises from the inefficient exploration of the token space in existing methods, such as Greedy Coordinate...

MAI-2024-0026

Large Language Models LLMs are susceptible to a sophisticated jailbreaking attack known as "MathPrompt." This technique exploits the models' proficiency in processing symbolic mathematics to circumvent inherent safety protocols. By embedding harmful natural language prompts within mathematically...

MAI-2024-0066

This vulnerability pertains to the safety alignment mechanisms of large language models LLMs, facilitating a "weak-to-strong" jailbreaking attack. The attack methodology involves employing a smaller, adversarially trained LLM, referred to as "unsafe," to manipulate the decoding probabilities of a...

MAI-2024-0058

Large Language Models LLMs are susceptible to jailbreaking attacks through the use of adversarially crafted suffixes. The AmpleGCG attack methodically generates a multitude of diverse and effective suffixes that successfully bypass safety protocols in both open-source and proprietary LLMs. This...