FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers

Monolithic Firmware is widespread. Unsurprisingly, fuzz testing firmware is an active research field with new advances addressing the unique challenges in the domain. However, understanding and evaluating improvements by deriving metrics such as code coverage and unique crashes are problematic,...

Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation

Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models LLMs. However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive...

SafeGenBench: a Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

The code generation capabilities of large language modelsLLMs have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark...

SoK: Data Reconstruction Attacks against Machine Learning Models: Definition, Metrics, and Benchmark

Data reconstruction attacks, which aim to recover the training dataset of a target model with limited access, have gained increasing attention in recent years. However, there is currently no consensus on a formal definition of data reconstruction attacks or appropriate evaluation metrics for...

Benchmarking Poisoning Attacks against Retrieval-Augmented Generation

Retrieval-Augmented Generation RAG has proven effective in mitigating hallucinations in large language models by incorporating external knowledge during inference. However, this integration introduces new security vulnerabilities, particularly to poisoning attacks. Although prior work has explore...

GenoArmory: a Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models

We propose the first unified adversarial attack benchmark for Genomic Foundation Models GFMs, named GenoArmory. Unlike existing GFM benchmarks, GenoArmory offers the first comprehensive evaluation framework to systematically assess the vulnerability of GFMs to adversarial attacks. Methodologicall...

Gaming Platforms as an attack vector against remote systems

Little more than a year ago I wrote about the possibility to attack gaming platform to compromise large audience of gamers in stealthy way, the access to millions of machines represent a dream for every attackers and I hypnotized its repercussion in cyber warfare domains. Gaming platform are...