Attack and Defense Techniques in Large Language Models: a Survey and New Perspectives

Large Language Models LLMs have become central to numerous natural language processing tasks, but their vulnerabilities present significant security and ethical challenges. This systematic survey explores the evolving landscape of attack and defense techniques in LLMs. We classify attacks into...

Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the...

How to Backdoor the Knowledge Distillation

Knowledge distillation has become a cornerstone in modern machine learning systems, celebrated for its ability to transfer knowledge from a large, complex teacher model to a more efficient student model. Traditionally, this process is regarded as secure, assuming the teacher model is clean. This...

Mitigating the Structural Bias in Graph Adversarial Defenses

In recent years, graph neural networks GNNs have shown great potential in addressing various graph structure-related downstream tasks. However, recent studies have found that current GNNs are susceptible to malicious adversarial attacks. Given the inevitable presence of adversarial attacks in the...

Quantifying the Noise of Structural Perturbations on Graph Adversarial Attacks

Graph neural networks have been widely utilized to solve graph-related tasks because of their strong learning power in utilizing the local information of neighbors. However, recent studies on graph adversarial attacks have proven that current graph neural networks are not robust against malicious...

The Dark Side of Digital Twins: Adversarial Attacks on AI-Driven Water Forecasting

Digital twins DTs are improving water distribution systems by using real-time data, analytics, and prediction models to optimize operations. This paper presents a DT platform designed for a Spanish water supply network, utilizing Long Short-Term Memory LSTM networks to predict water consumption...

AGATE: Stealthy Black-Box Watermarking for Multimodal Model Copyright Protection

Recent advancement in large-scale Artificial Intelligence AI models offering multimodal services have become foundational in AI systems, making them prime targets for model theft. Existing methods select Out-of-Distribution OoD data as backdoor watermarks and retrain the original model for...

Graph of Attacks: Improved Black-Box and Interpretable Jailbreaks for LLMs

The challenge of ensuring Large Language Models LLMs align with societal standards is of increasing interest, as these models are still prone to adversarial jailbreaks that bypass their safety mechanisms. Identifying these vulnerabilities is crucial for enhancing the robustness of LLMs against su...

PICO: Secure Transformers Via Robust Prompt Isolation and Cybersecurity Oversight

We propose a robust transformer architecture designed to prevent prompt injection attacks and ensure secure, reliable response generation. Our PICO Prompt Isolation and Cybersecurity Oversight framework structurally separates trusted system instructions from untrusted user inputs through dual...

Adversarial Attacks on LLM-As-A-Judge Systems: Insights from Prompt Injections

LLM as judge systems used to assess text quality code correctness and argument strength are vulnerable to prompt injection attacks. We introduce a framework that separates content author attacks from system prompt attacks and evaluate five models Gemma 3.27B Gemma 3.4B Llama 3.2 3B GPT 4 and Clau...

Performance of Machine Learning Classifiers for Anomaly Detection in Cyber Security Applications

This work empirically evaluates machine learning models on two imbalanced public datasets KDDCUP99 and Credit Card Fraud 2013. The method includes data preparation, model training, and evaluation, using an 80/20 train/test split. Models tested include eXtreme Gradient Boosting XGB, Multi Layer...

New whitepaper outlines the taxonomy of failure modes in AI agents

We are releasing a taxonomy of failure modes in AI agents to help security professionals and machine learning engineers think through how AI systems can fail and design them with safety and security in mind. The taxonomy continues Microsoft AI Red Team's work to lead the creation of systematizati...

Cluster-Aware Attacks on Graph Watermarks

Data from domains such as social networks, healthcare, finance, and cybersecurity can be represented as graph-structured information. Given the sensitive nature of this data and their frequent distribution among collaborators, ensuring secure and attributable sharing is essential. Graph...

Evaluating the Vulnerability of ML-Based Ethereum Phishing Detectors to Single-Feature Adversarial Perturbations

This paper explores the vulnerability of machine learning models to simple single-feature adversarial attacks in the context of Ethereum fraudulent transaction detection. Through comprehensive experimentation, we investigate the impact of various adversarial attack strategies on model performance...

Enhancing Variational Autoencoders with Smooth Robust Latent Encoding

Variational Autoencoders VAEs have played a key role in scaling up diffusion-based generative models, as in Stable Diffusion, yet questions regarding their robustness remain largely underexplored. Although adversarial training has been an established technique for enhancing robustness in predicti...

Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: a Theoretical Framework and Algorithmic Instantiation

Whitepaper called Seeking Flat Minima Over Diverse Surrogates For Improved Adversarial Transferability: A Theoretical Framework And Algorithmic Instantiation...

AiXamine: Simplified LLM Safety and Security

Evaluating Large Language Models LLMs for safety and security remains a complex task, often requiring users to navigate a fragmented landscape of ad hoc benchmarks, datasets, metrics, and reporting formats. To address this challenge, we present aiXamine, a comprehensive black-box evaluation...

Adversarial Observations in Weather Forecasting

AI-based systems, such as Google's GenCast, have recently redefined the state of the art in weather forecasting, offering more accurate and timely predictions of both everyday weather and extreme events. While these systems are on the verge of replacing traditional meteorological methods, they al...

How Private Is Your Attention? Bridging Privacy with In-Context Learning

In-context learning ICL-the ability of transformer-based models to perform new tasks from examples provided at inference time-has emerged as a hallmark of modern language models. While recent works have investigated the mechanisms underlying ICL, its feasibility under formal privacy constraints...

Property-Preserving Hashing for $\Ell_1$-Distance Predicates: Applications to Countering Adversarial Input Attacks

Perceptual hashing is used to detect whether an input image is similar to a reference image with a variety of security applications. Recently, they have been shown to succumb to adversarial input attacks which make small imperceptible changes to the input image yet the hashing algorithm does not...