VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents

Computer-Use Agents CUAs with full system access enable powerful task automation but pose significant security and privacy risks due to their ability to manipulate files, access user data, and execute arbitrary commands. While prior work has focused on browser-based agents and HTML-level attacks,...

Con Instruction: Universal Jailbreaking of Multimodal Large Language Models Via Non-Textual Modalities

Existing attacks against multimodal language models MLLMs primarily communicate instructions through text accompanied by adversarial images. In contrast, we exploit the capabilities of MLLMs to interpret non-textual instructions, specifically, adversarial images or audio generated by our novel...

Data Flows in You: Benchmarking and Improving Static Data-Flow Analysis on Binary Executables

Data-flow analysis is a critical component of security research. Theoretically, accurate data-flow analysis in binary executables is an undecidable problem, due to complexities of binary code. Practically, many binary analysis engines offer some data-flow analysis capability, but we lack...

A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?

We introduce the Robust Audio Watermarking Benchmark RAW-Bench, a benchmark for evaluating deep learning-based audio watermarking methods with standardized and systematic comparisons. To simulate real-world usage, we introduce a comprehensive audio attack pipeline with various distortions such as...

VideoMarkBench: Benchmarking Robustness of Video Watermarking

The rapid development of video generative models has led to a surge in highly realistic synthetic videos, raising ethical concerns related to disinformation and copyright infringement. Recently, video watermarking has been proposed as a mitigation strategy by embedding invisible marks into...

Capability-Based Scaling Laws for LLM Red-Teaming

As large language models grow in capability and agency, identifying vulnerabilities through red-teaming becomes vital for safe deployment. However, traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a weak-to-strong problem, where target models surpass...

PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks

Large language models LLMs have achieved remarkable capabilities but remain vulnerable to adversarial prompts known as jailbreaks, which can bypass safety alignment and elicit harmful outputs. Despite growing efforts in LLM safety research, existing evaluations are often fragmented, focused on...

Fixing 7,400 Bugs for 1$: Cheap Crash-Site Program Repair

The rapid advancement of bug-finding techniques has led to the discovery of more vulnerabilities than developers can reasonably fix, creating an urgent need for effective Automated Program Repair APR methods. However, the complexity of modern bugs often makes precise root cause analysis difficult...

LAMDA: a Longitudinal Android Malware Benchmark for Concept Drift Analysis

Machine learning ML-based malware detection systems often fail to account for the dynamic nature of real-world training and test data distributions. In practice, these distributions evolve due to frequent changes in the Android ecosystem, adversarial development of new malware families, and the...

Benchmarking Poisoning Attacks against Retrieval-Augmented Generation

Retrieval-Augmented Generation RAG has proven effective in mitigating hallucinations in large language models by incorporating external knowledge during inference. However, this integration introduces new security vulnerabilities, particularly to poisoning attacks. Although prior work has explore...

CVE-2024-27508

Atheme 7.2.12 contains a memory leak vulnerability in /atheme/src/crypto-benchmark/main.c...

CVE-2024-31360

Cross-Site Request Forgery CSRF vulnerability in Coded Commerce, LLC Benchmark Email Lite.This issue affects Benchmark Email Lite: from n/a through 4.1...

CVE-2024-31922

Cross-Site Request Forgery CSRF vulnerability in Anton Aleksandrov WordPress Hosting Benchmark tool.This issue affects WordPress Hosting Benchmark tool: from n/a through 1.3.6...

CVE-2023-32114

SAP NetWeaver Change and Transport System - versions 702, 731, 740, 750, 751, 752, 753, 754, 755, 756, 757, allows an authenticated user with admin privileges to maliciously run a benchmark program repeatedly in intent to slowdown or make the server unavailable which may lead to a limited impact ...

Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study

Rapid deployment of vision-language models VLMs magnifies safety risks, yet most evaluations rely on artificial images. This study asks: How safe are current VLMs when confronted with meme images that ordinary users share? To investigate this question, we introduce MemeSafetyBench, a...

Evaluating the Efficacy of LLM Safety Solutions : the Palit Benchmark Dataset

Large Language Models LLMs are increasingly integrated into critical systems in industries like healthcare and finance. Users can often submit queries to LLM-enabled chatbots, some of which can enrich responses with information retrieved from internal databases storing sensitive data. This gives...

Improving LLM Outputs against Jailbreak Attacks with Expert Model Integration

Using LLMs in a production environment presents security challenges that include vulnerabilities to jailbreaks and prompt injections, which can result in harmful outputs for humans or the enterprise. The challenge is amplified when working within a specific domain, as topics generally accepted fo...

Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models LLMs offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue...

WASP: Benchmarking Web Agent Security against Prompt Injection Attacks

Autonomous UI agents powered by AI have tremendous potential to boost human productivity by automating routine tasks such as filing taxes and paying bills. However, a major challenge in unlocking their full potential is security, which is exacerbated by the agent's ability to take action on their...

GenoArmory: a Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models

We propose the first unified adversarial attack benchmark for Genomic Foundation Models GFMs, named GenoArmory. Unlike existing GFM benchmarks, GenoArmory offers the first comprehensive evaluation framework to systematically assess the vulnerability of GFMs to adversarial attacks. Methodologicall...