DecipherGuard: Understanding and Deciphering Jailbreak Prompts for a Safer Deployment of Intelligent Software Systems

Intelligent software systems powered by Large Language Models LLMs are increasingly deployed in critical sectors, raising concerns about their safety during runtime. Through an industry-academic collaboration when deploying an LLM-powered virtual customer assistant, a critical software engineerin...

Amazon Nova AI Challenge -- Trusted AI: Advancing Secure, AI-Assisted Software Development

AI systems for software development are rapidly gaining prominence, yet significant challenges remain in ensuring their safety. To address this, Amazon launched the Trusted AI track of the Amazon Nova AI Challenge, a global competition among 10 university teams to drive advances in secure AI. In...

Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security

As large language models LLMs increasingly integrate native code interpreters, they enable powerful real-time execution capabilities, substantially expanding their utility. However, such integrations introduce potential system-level cybersecurity threats, fundamentally different from prompt-based...

SecurityLingua: Efficient Defense of LLM Jailbreak Attacks Via Security-Aware Prompt Compression

Large language models LLMs have achieved widespread adoption across numerous applications. However, many LLMs are vulnerable to malicious attacks even after safety alignment. These attacks typically bypass LLMs' safety guardrails by wrapping the original malicious instructions inside adversarial...

Rubber Mallet: a Study of High Frequency Localized Bit Flips and Their Impact on Security

The increasing density of modern DRAM has heightened its vulnerability to Rowhammer attacks, which induce bit flips by repeatedly accessing specific memory rows. This paper presents an analysis of bit flip patterns generated by advanced Rowhammer techniques that bypass existing hardware defenses...

SoK: Evaluating Jailbreak Guardrails for Large Language Models

Large Language Models LLMs have achieved remarkable progress, but their deployment has exposed critical vulnerabilities, particularly to jailbreak attacks that circumvent safety mechanisms. Guardrails--external defense mechanisms that monitor and control LLM interaction--have emerged as a promisi...

Why LLM Safety Guardrails Collapse after Fine-Tuning: a Similarity Analysis between Alignment and Fine-Tuning Datasets

Recent advancements in large language models LLMs have underscored their vulnerability to safety alignment jailbreaks, particularly when subjected to downstream fine-tuning. However, existing mitigation strategies primarily focus on reactively addressing jailbreak incidents after safety guardrail...

CVE-2024-6961

RAIL documents are an XML-based format invented by Guardrails AI to enforce formatting checks on LLM outputs. Guardrails users that consume RAIL documents from external sources are vulnerable to XXE, which may cause leakage of internal file data via the SYSTEM entity...

SudoLLM : on Multi-Role Alignment of Language Models

User authorization-based access privileges are a key feature in many safety-critical systems, but have thus far been absent from the large language model LLM realm. In this work, drawing inspiration from such access control systems, we introduce sudoLLM, a novel framework that results in multi-ro...

Security Steerability Is All You Need

The adoption of Generative AI GenAI in various applications inevitably comes with expanding the attack surface, combining new security threats along with the traditional ones. Consequently, numerous research and industrial initiatives aim to mitigate these security threats in GenAI by developing...

Hoist with His Own Petard: Inducing Guardrails to Facilitate Denial-Of-Service Attacks on Retrieval-Augmented Generation of LLMs

Whitepaper called Hoist With His Own Petard: Inducing Guardrails To Facilitate Denial-Of-Service Attacks On Retrieval-Augmented Generation Of LLMs...

Various GPT services are vulnerable to two systemic jailbreaks, allows for bypass of safety guardrails

Overview Two systemic jailbreaks, affecting a number of generative AI services, were discovered. These jailbreaks can result in the bypass of safety protocols and allow an attacker to instruct the corresponding LLM to provide illicit or dangerous content. The first jailbreak, called “Inception,” ...

DualBreach: Efficient Dual-Jailbreaking Via Target-Driven Initialization and Multi-Target Optimization

Recent research has focused on exploring the vulnerabilities of Large Language Models LLMs, aiming to elicit harmful and/or sensitive content from LLMs. However, due to the insufficient research on dual-jailbreaking -- attacks targeting both LLMs and Guardrails, the effectiveness of existing...

AWS VDP: Bedrock Guardrails Evasion with Prompt Formatting

Description Greetings, my name is ██████ and I am a Director here at NR Labs. We recently completed disclosure of this vulnerability by working with ████ and the AWS Security team. We are submitting this issue to the AWS VDP to create an official record of the issue with AWS in preparation for a...

CVE-2024-45858

An arbitrary code execution vulnerability exists in versions 0.2.9 up to 0.5.10 of the Guardrails AI Guardrails framework because of the way it validates XML files. If a victim user loads a maliciously crafted XML file containing Python code, the code will be passed to an eval function, causing i...

Race Condition Attacks against LLMs

These are two attacks against the system components surrounding LLMs: We propose that LLM Flowbreaking, following jailbreaking and prompt injection, joins as the third on the growing list of LLM attack types. Flowbreaking is less about whether prompt or response guardrails can be bypassed, and mo...

Researchers Reveal 'Deceptive Delight' Method to Jailbreak AI Models

Cybersecurity researchers have shed light on a new adversarial technique that could be used to jailbreak large language models LLMs during the course of an interactive conversation by sneaking in an undesirable instruction between benign ones. The approach has been codenamed Deceptive Delight by...

A Single Cloud Compromise Can Feed an Army of AI Sex Bots

Organizations that get relieved of credentials to their cloud environments can quickly find themselves part of a disturbing new trend: Cybercriminals using stolen cloud credentials to operate and resell sexualized AI-powered chat services. Researchers say these illicit chat bots, which use custom...

Guardrails has an arbitrary code execution vulnerability

An arbitrary code execution vulnerability exists in versions 0.2.9 up to 0.5.10 of the Guardrails AI Guardrails framework because of the way it validates XML files. If a victim user loads a maliciously crafted XML file containing Python code, the code will be passed to an eval function, causing i...

GHSA-W392-75Q8-VR67 Guardrails has an arbitrary code execution vulnerability

An arbitrary code execution vulnerability exists in versions 0.2.9 up to 0.5.10 of the Guardrails AI Guardrails framework because of the way it validates XML files. If a victim user loads a maliciously crafted XML file containing Python code, the code will be passed to an eval function, causing i...