System Prompt Extraction Attacks and Defenses in Large Language Models

The system prompt in Large Language Models LLMs plays a pivotal role in guiding model behavior and response generation. Often containing private configuration details, user roles, and operational instructions, the system prompt has become an emerging attack target. Recent studies have shown that...

Semantic-Preserving Adversarial Attacks on LLMs: an Adaptive Greedy Binary Search Approach

Large Language Models LLMs increasingly rely on automatic prompt engineering in graphical user interfaces GUIs to refine user inputs and enhance response accuracy. However, the diversity of user requirements often leads to unintended misinterpretations, where automated optimizations distort...

PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks

Large language models LLMs have achieved remarkable capabilities but remain vulnerable to adversarial prompts known as jailbreaks, which can bypass safety alignment and elicit harmful outputs. Despite growing efforts in LLM safety research, existing evaluations are often fragmented, focused on...

Phare: a Safety Probe for Large Language Models

Ensuring the safety of large language models LLMs is critical for responsible deployment, yet existing evaluations often prioritize performance over identifying failure modes. We introduce Phare, a multilingual diagnostic framework to probe and evaluate LLM behavior across three critical...

CoTGuard: Using Chain-Of-Thought Triggering for Copyright Protection in Multi-Agent LLM Systems

As large language models LLMs evolve into autonomous agents capable of collaborative reasoning and task execution, multi-agent LLM systems have emerged as a powerful paradigm for solving complex problems. However, these systems pose new challenges for copyright protection, particularly when...

Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models

State-of-the-art membership inference attacks MIAs typically require training many reference models, making it difficult to scale these attacks to large pre-trained language models LLMs. As a result, prior research has either relied on weaker attacks that avoid training reference models e.g.,...

One Model Transfer to All: on Robust Jailbreak Prompts Generation against LLMs

Safety alignment in large language models LLMs is increasingly compromised by jailbreak attacks, which can manipulate these models to generate harmful or unintended content. Investigating these attacks is crucial for uncovering model vulnerabilities. However, many existing jailbreak strategies fa...

Invisible Tokens, Visible Bills: the Urgent Need to Audit Hidden Operations in Opaque LLM Services

Whitepaper called Invisible Tokens, Visible Bills: The Urgent Need To Audit Hidden Operations In Opaque LLM Services...

ACSE-Eval: Can LLMs Threat Model Real-World Cloud Infrastructure?

While Large Language Models have shown promise in cybersecurity applications, their effectiveness in identifying security threats within cloud deployments remains unexplored. This paper introduces AWS Cloud Security Engineering Eval, a novel dataset for evaluating LLMs cloud security threat...

Mapping the Future of AI Security

AI security is one of the most pressing challenges facing the world today. Artificial intelligence is extraordinarily powerful, and, especially considering the advent of Agentic AI, growing more so by the day. But it is for this reason that securing it is so important. AI handles massive amounts ...

Advancing Security with Digital Twins: a Comprehensive Survey

The proliferation of electronic devices has greatly transformed every aspect of human life, such as communication, healthcare, transportation, and energy. Unfortunately, the global electronics supply chain is vulnerable to various attacks, including piracy of intellectual properties, tampering,...

When Safety Detectors Aren'T Enough: a Stealthy and Effective Jailbreak Attack on LLMs Via Steganographic Techniques

Jailbreak attacks pose a serious threat to large language models LLMs by bypassing built-in safety mechanisms and leading to harmful outputs. Studying these attacks is crucial for identifying vulnerabilities and improving model security. This paper presents a systematic survey of jailbreak method...

CAIN: Hijacking LLM-Humans Conversations Via a Two-Stage Malicious System Prompt Generation and Refining Framework

Large language models LLMs have advanced many applications, but are also known to be vulnerable to adversarial attacks. In this work, we introduce a novel security threat: hijacking AI-human conversations by manipulating LLMs' system prompts to produce malicious answers only to specific targeted...

Mitigating Fine-Tuning Risks in LLMs Via Safety-Aware Probing Optimization

The significant progress of large language models LLMs has led to remarkable achievements across numerous applications. However, their ability to generate harmful content has sparked substantial safety concerns. Despite the implementation of safety alignment techniques during the pre-training...

Unlearning Isn'T Deletion: Investigating Reversibility of Machine Unlearning in LLMs

Unlearning in large language models LLMs is intended to remove the influence of specific data, yet current evaluations rely heavily on token-level metrics such as accuracy and perplexity. We show that these metrics can be misleading: models often appear to forget, but their original behavior can ...

MTSA: Multi-Turn Safety Alignment for LLMs through Multi-Round Red-Teaming

Whitepaper called MTSA: Multi-Turn Safety Alignment For LLMs Through Multi-Round Red-Teaming...

CoTSRF: Utilize Chain of Thought As Stealthy and Robust Fingerprint of Large Language Models

Despite providing superior performance, open-source large language models LLMs are vulnerable to abusive usage. To address this issue, recent works propose LLM fingerprinting methods to identify the specific source LLMs behind suspect applications. However, these methods fail to provide stealthy...

DuFFin: a Dual-Level Fingerprinting Framework for LLMs IP Protection

Whitepaper called DuFFin: A Dual-Level Fingerprinting Framework For LLMs IP Protection...

Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: an Empirical Study on Popular Open-Source Projects

Command injection vulnerabilities are a significant security threat in dynamic languages like Python, particularly in widely used open-source projects where security issues can have extensive impact. With the proven effectiveness of Large Language ModelsLLMs in code-related tasks, such as testing...

One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems

Large Language Models LLMs enhanced with Retrieval-Augmented Generation RAG have shown improved performance in generating accurate responses. However, the dependence on external knowledge bases introduces potential security vulnerabilities, particularly when these knowledge bases are publicly...