Google Adds Multi-Layered Defenses to Secure GenAI from Prompt Injection Attacks

Google has revealed the various safety measures that are being incorporated into its generative artificial intelligence AI systems to mitigate emerging attack vectors like indirect prompt injections and improve the overall security posture for agentic AI systems. "Unlike direct prompt injections,...

Enhancing Security in LLM Applications: a Performance Evaluation of Early Detection Systems

Prompt injection threatens novel applications that emerge from adapting LLMs for various user tasks. The newly developed LLM-based software applications become more ubiquitous and diverse. However, the threat of prompt injection attacks undermines the security of these systems as the mitigation a...

AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

We introduce AIRTBench, an AI red teaming benchmark for evaluating language models' ability to autonomously discover and exploit Artificial Intelligence and Machine Learning AI/ML security vulnerabilities. The benchmark consists of 70 realistic black-box capture-the-flag CTF challenges from the...

CVE-2025-49150

Cursor is a code editor built for programming with AI. Prior to 0.51.0, by default, the setting json.schemaDownload.enable was set to True. This means that by writing a JSON file, an attacker can trigger an arbitrary HTTP GET request that does not require user confirmation. Since the Cursor Agent...

AgentVigil: Generic Black-Box Red-Teaming for Indirect Prompt Injection against LLM Agents

The strong planning and reasoning capabilities of Large Language Models LLMs have fostered the development of agent-based systems capable of leveraging external tools and interacting with increasingly complex environments. However, these powerful features also introduce a critical security risk:...

LLMail-Inject: a Dataset from a Realistic Adaptive Prompt Injection Challenge

Indirect Prompt Injection attacks exploit the inherent limitation of Large Language Models LLMs to distinguish between instructions and data in their inputs. Despite numerous defense proposals, the systematic evaluation against adaptive adversaries remains limited, even when successful attacks ca...

Evaluation Empirique De La Sécurisation Et De L'Alignement De ChatGPT Et Gemini: Analyse Comparative Des Vulnérabilités Par Expérimentations De Jailbreaks

Large Language models LLMs are transforming digital usage, particularly in text generation, image creation, information retrieval and code development. ChatGPT, launched by OpenAI in November 2022, quickly became a reference, prompting the emergence of competitors such as Google's Gemini. However...

Sentinel: SOTA Model to Protect against Prompt Injections

Large Language Models LLMs are increasingly powerful but remain vulnerable to prompt injection attacks, where malicious inputs cause the model to deviate from its intended instructions. This paper introduces Sentinel, a novel detection model, qualifire/prompt-injection-sentinel, based on the...

VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents

Computer-Use Agents CUAs with full system access enable powerful task automation but pose significant security and privacy risks due to their ability to manipulate files, access user data, and execute arbitrary commands. While prior work has focused on browser-based agents and HTML-level attacks,...

ATAG: AI-Agent Application Threat Assessment with Attack Graphs

Evaluating the security of multi-agent systems MASs powered by large language models LLMs is challenging, primarily because of the systems' complex internal dynamics and the evolving nature of LLM vulnerabilities. Traditional attack graph AG methods often lack the specific capabilities to model...

Security Concerns for Large Language Models: a Survey

Large Language Models LLMs such as GPT-4 and its recent iterations, Google's Gemini, Anthropic's Claude 3 models, and xAI's Grok have caused a revolution in natural language processing, but their capabilities also introduce new security vulnerabilities. In this survey, we provide a comprehensive...

Adversarial Threat Vectors and Risk Mitigation for Retrieval-Augmented Generation Systems

Retrieval-Augmented Generation RAG systems, which integrate Large Language Models LLMs with external knowledge sources, are vulnerable to a range of adversarial attack vectors. This paper examines the importance of RAG systems through recent industry adoption trends and identifies the prominent...

Hijacking Large Language Models Via Adversarial In-Context Learning

In-context learning ICL has emerged as a powerful paradigm leveraging LLMs for specific downstream tasks by utilizing labeled examples as demonstrations demos in the preconditioned prompts. Despite its promising performance, crafted adversarial attacks pose a notable threat to the robustness of...

CVE-2024-45989

Monica AI Assistant desktop application v2.3.0 is vulnerable to Exposure of Sensitive Information to an Unauthorized Actor. A prompt injection allows an attacker to modify chatbot answer with an unloaded image that exfiltrates the user's sensitive chat data of the current session to a malicious...

CVE-2024-7110

An issue was discovered in GitLab EE affecting all versions starting 17.0 to 17.1.6, 17.2 prior to 17.2.4, and 17.3 prior to 17.3.1 allows an attacker to execute arbitrary command in a victim's pipeline through prompt injection...

CVE-2024-7042

A vulnerability in the GraphCypherQAChain class of langchain-ai/langchainjs versions 0.2.5 and all versions with this class allows for prompt injection, leading to SQL injection. This vulnerability permits unauthorized data manipulation, data exfiltration, denial of service DoS by deleting all...

CVE-2024-25639

Khoj is an application that creates personal AI agents. The Khoj Obsidian, Desktop and Web clients inadequately sanitize the AI model's response and user inputs. This can trigger Cross Site Scripting XSS via Prompt Injection from untrusted documents either indexed by the user on Khoj or read by...

CVE-2024-4099

An issue has been discovered in GitLab EE affecting all versions starting from 16.0 prior to 17.2.8, from 17.3 prior to 17.3.4, and from 17.4 prior to 17.4.1. An AI feature was found to read unsanitized content in a way that could have allowed an attacker to hide prompt injection...

CVE-2024-48144

A prompt injection vulnerability in the chatbox of Fusion Chat Chat AI Assistant Ask Me Anything v1.2.4.0 allows attackers to access and exfiltrate all previous and subsequent chat data between the user and the AI assistant via a crafted message...

CVE-2024-48139

A prompt injection vulnerability in the chatbox of Blackbox AI v1.3.95 allows attackers to access and exfiltrate all previous and subsequent chat data between the user and the AI assistant via a crafted message...