Towards Secure MLOps: Surveying Attacks, Mitigation Strategies, and Research Challenges

The rapid adoption of machine learning ML technologies has driven organizations across diverse sectors to seek efficient and reliable methods to accelerate model development-to-deployment. Machine Learning Operations MLOps has emerged as an integrative approach addressing these requirements by...

Hijacking Large Language Models Via Adversarial In-Context Learning

In-context learning ICL has emerged as a powerful paradigm leveraging LLMs for specific downstream tasks by utilizing labeled examples as demonstrations demos in the preconditioned prompts. Despite its promising performance, crafted adversarial attacks pose a notable threat to the robustness of...

Disrupting Vision-Language Model-Driven Navigation Services Via Adversarial Object Fusion

We present Adversarial Object Fusion AdvOF, a novel attack framework targeting vision-and-language navigation VLN agents in service-oriented environments by generating adversarial 3D objects. While foundational models like Large Language Models LLMs and Vision Language Models VLMs have enhanced...

Fooling the Watchers: Breaking AIGC Detectors Via Semantic Prompt Attacks

The rise of text-to-image T2I models has enabled the synthesis of photorealistic human portraits, raising serious concerns about identity misuse and the robustness of AIGC detectors. In this work, we propose an automated adversarial prompt generation framework that leverages a grammar tree...

Preventing Adversarial AI Attacks against Autonomous Situational Awareness: a Maritime Case Study

Adversarial artificial intelligence AI attacks pose a significant threat to autonomous transportation, such as maritime vessels, that rely on AI components. Malicious actors can exploit these systems to deceive and manipulate AI-driven operations. This paper addresses three critical research...

CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models

Retrieval-Augmented Generation RAG enhances large language models LLMs by incorporating external knowledge, but its openness introduces vulnerabilities that can be exploited by poisoning attacks. Existing poisoning methods for RAG systems have limitations, such as poor generalization and lack of...

Efficient and Stealthy Jailbreak Attacks Via Adversarial Prompt Distillation from LLMs to SLMs

Attacks on large language models LLMs in jailbreaking scenarios raise many security and ethical issues. Current jailbreak attack methods face problems such as low efficiency, high computational cost, and poor cross-model adaptability and versatility, which make it difficult to cope with the rapid...

Novel Loss-Enhanced Universal Adversarial Patches for Sustainable Speaker Privacy

Deep learning voice models are commonly used nowadays, but the safety processing of personal data, such as human identity and speech content, remains suspicious. To prevent malicious user identification, speaker anonymization methods were proposed. Current methods, particularly based on universal...

Semantic-Preserving Adversarial Attacks on LLMs: an Adaptive Greedy Binary Search Approach

Large Language Models LLMs increasingly rely on automatic prompt engineering in graphical user interfaces GUIs to refine user inputs and enhance response accuracy. However, the diversity of user requirements often leads to unintended misinterpretations, where automated optimizations distort...

MultiPhishGuard: an LLM-Based Multi-Agent System for Phishing Email Detection

Phishing email detection faces critical challenges from evolving adversarial tactics and heterogeneous attack patterns. Traditional detection methods, such as rule-based filters and denylists, often struggle to keep pace with these evolving tactics, leading to false negatives and compromised...

RADEP: a Resilient Adaptive Defense Framework against Model Extraction Attacks

Machine Learning as a Service MLaaS enables users to leverage powerful machine learning models through cloud-based APIs, offering scalability and ease of deployment. However, these services are vulnerable to model extraction attacks, where adversaries repeatedly query the application programming...

ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast \& Slow Reasoning for Robust Agent Defense

LLM Agents are becoming central to intelligent systems. However, their deployment raises serious safety concerns. Existing defenses largely rely on "Safety Checks", which struggle to capture the complex semantic risks posed by harmful user inputs or unsafe agent behaviors - creating a significant...

Mal-D2GAN: Double-Detector Based GAN for Malware Generation

Machine learning ML has been developed to detect malware in recent years. Most researchers focused their efforts on improving the detection performance but ignored the robustness of the ML models. In addition, many machine learning algorithms are very vulnerable to intentional attacks. To solve...

Revisiting Adversarial Perception Attacks and Defense Methods on Autonomous Driving Systems

Autonomous driving systems ADS increasingly rely on deep learning-based perception models, which remain vulnerable to adversarial attacks. In this paper, we revisit adversarial attacks and defense methods, focusing on road sign recognition and lead object detection and prediction e.g., relative...

CAIN: Hijacking LLM-Humans Conversations Via a Two-Stage Malicious System Prompt Generation and Refining Framework

Large language models LLMs have advanced many applications, but are also known to be vulnerable to adversarial attacks. In this work, we introduce a novel security threat: hijacking AI-human conversations by manipulating LLMs' system prompts to produce malicious answers only to specific targeted...

Privacy-Aware Cyberterrorism Network Analysis Using Graph Neural Networks and Federated Learning

Cyberterrorism poses a formidable threat to digital infrastructures, with increasing reliance on encrypted, decentralized platforms that obscure threat actor activity. To address the challenge of analyzing such adversarial networks while preserving the privacy of distributed intelligence data, we...

SuperPure: Efficient Purification of Localized and Distributed Adversarial Patches Via Super-Resolution GAN Models

As vision-based machine learning models are increasingly integrated into autonomous and cyber-physical systems, concerns about physical adversarial patch attacks are growing. While state-of-the-art defenses can achieve certified robustness with minimal impact on utility against highly-concentrate...

Harry Potter Is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models Via Automated Adversarial Prompting

This work presents LURK Latent UnleaRned Knowledge, a novel framework that probes for hidden retained knowledge in unlearned LLMs through adversarial suffix prompting. LURK automatically generates adversarial prompt suffixes designed to elicit residual knowledge about the Harry Potter domain, a...

Scalable Defense against In-The-Wild Jailbreaking Attacks with Safety Context Retrieval

Large Language Models LLMs are known to be vulnerable to jailbreaking attacks, wherein adversaries exploit carefully engineered prompts to induce harmful or unethical responses. Such threats have raised critical concerns about the safety and reliability of LLMs in real-world deployment. While...

Alignment under Pressure: the Case for Informed Adversaries When Evaluating LLM Defenses

Large language models LLMs are rapidly deployed in real-world applications ranging from chatbots to agentic systems. Alignment is one of the main approaches used to defend against attacks such as prompt injection and jailbreaks. Recent defenses report near-zero Attack Success Rates ASR even again...