Invisible Tokens, Visible Bills: the Urgent Need to Audit Hidden Operations in Opaque LLM Services

Whitepaper called Invisible Tokens, Visible Bills: The Urgent Need To Audit Hidden Operations In Opaque LLM Services...

ACSE-Eval: Can LLMs Threat Model Real-World Cloud Infrastructure?

While Large Language Models have shown promise in cybersecurity applications, their effectiveness in identifying security threats within cloud deployments remains unexplored. This paper introduces AWS Cloud Security Engineering Eval, a novel dataset for evaluating LLMs cloud security threat...

Mapping the Future of AI Security

AI security is one of the most pressing challenges facing the world today. Artificial intelligence is extraordinarily powerful, and, especially considering the advent of Agentic AI, growing more so by the day. But it is for this reason that securing it is so important. AI handles massive amounts ...

When Safety Detectors Aren'T Enough: a Stealthy and Effective Jailbreak Attack on LLMs Via Steganographic Techniques

Jailbreak attacks pose a serious threat to large language models LLMs by bypassing built-in safety mechanisms and leading to harmful outputs. Studying these attacks is crucial for identifying vulnerabilities and improving model security. This paper presents a systematic survey of jailbreak method...

Advancing Security with Digital Twins: a Comprehensive Survey

The proliferation of electronic devices has greatly transformed every aspect of human life, such as communication, healthcare, transportation, and energy. Unfortunately, the global electronics supply chain is vulnerable to various attacks, including piracy of intellectual properties, tampering,...

CAIN: Hijacking LLM-Humans Conversations Via a Two-Stage Malicious System Prompt Generation and Refining Framework

Large language models LLMs have advanced many applications, but are also known to be vulnerable to adversarial attacks. In this work, we introduce a novel security threat: hijacking AI-human conversations by manipulating LLMs' system prompts to produce malicious answers only to specific targeted...

Mitigating Fine-Tuning Risks in LLMs Via Safety-Aware Probing Optimization

The significant progress of large language models LLMs has led to remarkable achievements across numerous applications. However, their ability to generate harmful content has sparked substantial safety concerns. Despite the implementation of safety alignment techniques during the pre-training...

Unlearning Isn'T Deletion: Investigating Reversibility of Machine Unlearning in LLMs

Unlearning in large language models LLMs is intended to remove the influence of specific data, yet current evaluations rely heavily on token-level metrics such as accuracy and perplexity. We show that these metrics can be misleading: models often appear to forget, but their original behavior can ...

CoTSRF: Utilize Chain of Thought As Stealthy and Robust Fingerprint of Large Language Models

Despite providing superior performance, open-source large language models LLMs are vulnerable to abusive usage. To address this issue, recent works propose LLM fingerprinting methods to identify the specific source LLMs behind suspect applications. However, these methods fail to provide stealthy...

MTSA: Multi-Turn Safety Alignment for LLMs through Multi-Round Red-Teaming

Whitepaper called MTSA: Multi-Turn Safety Alignment For LLMs Through Multi-Round Red-Teaming...

DuFFin: a Dual-Level Fingerprinting Framework for LLMs IP Protection

Whitepaper called DuFFin: A Dual-Level Fingerprinting Framework For LLMs IP Protection...

Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMs

Whitepaper called Blind Spot Navigation: Evolutionary Discovery Of Sensitive Semantic Concepts For LVLMs...

Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study

Rapid deployment of vision-language models VLMs magnifies safety risks, yet most evaluations rely on artificial images. This study asks: How safe are current VLMs when confronted with meme images that ordinary users share? To investigate this question, we introduce MemeSafetyBench, a...

Scalable Defense against In-The-Wild Jailbreaking Attacks with Safety Context Retrieval

Large Language Models LLMs are known to be vulnerable to jailbreaking attacks, wherein adversaries exploit carefully engineered prompts to induce harmful or unethical responses. Such threats have raised critical concerns about the safety and reliability of LLMs in real-world deployment. While...

FragFake: a Dataset for Fine-Grained Detection of Edited Images with Vision Language Models

Fine-grained edited image detection of localized edits in images is crucial for assessing content authenticity, especially given that modern diffusion models and image editing methods can produce highly realistic manipulations. However, this domain faces three challenges: 1 Binary classifiers yie...

Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: an Empirical Study on Popular Open-Source Projects

Command injection vulnerabilities are a significant security threat in dynamic languages like Python, particularly in widely used open-source projects where security issues can have extensive impact. With the proven effectiveness of Large Language ModelsLLMs in code-related tasks, such as testing...

From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents

The growing adoption of large language models LLMs has led to a new paradigm in mobile computing--LLM-powered mobile AI agents--capable of decomposing and automating complex tasks directly on smartphones. However, the security implications of these agents remain largely unexplored. In this paper,...

Is Your Prompt Safe? Investigating Prompt Injection Attacks against Open-Source LLMs

Whitepaper called Is Your Prompt Safe? Investigating Prompt Injection Attacks Against Open-Source LLMs...

One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems

Large Language Models LLMs enhanced with Retrieval-Augmented Generation RAG have shown improved performance in generating accurate responses. However, the dependence on external knowledge bases introduces potential security vulnerabilities, particularly when these knowledge bases are publicly...

MorphMark: Flexible Adaptive Watermarking for Large Language Models

Watermarking by altering token sampling probabilities based on red-green list is a promising method for tracing the origin of text generated by large language models LLMs. However, existing watermark methods often struggle with a fundamental dilemma: improving watermark effectiveness the...