SEC-Bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

Large language models LLMs now support automated software security tasks, including vulnerability discovery and proof-of-concept PoC generation. Existing benchmarks do not faithfully evaluate LLMs in real-world bug hunting scenarios because they rely on fuzzing harnesses, target-specific...

n-days-poc-benchmark-and-dataset

ICS N-Day Vulnerability PoC Benchmark Suite A structured coll...

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Vibe coding is a new programming paradigm in which human engineers instruct large language model LLM agents to complete complex coding tasks with little supervision. Although it is increasingly adopted, are vibe coding outputs really safe to deploy in production? To answer this question, we propo...

Systematic Assessment of Cache Timing Vulnerabilities on RISC-V Processors

While interest in the open RISC-V instruction set architecture is growing, tools to assess the security of concrete processor implementations are lacking. There are dedicated tools and benchmarks for common microarchitectural side-channel vulnerabilities for popular processor families such as Int...

SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios

Large language model LLM powered code agents are rapidly transforming software engineering by automating tasks such as testing, debugging, and repairing, yet the security risks of their generated code have become a critical concern. Existing benchmarks have offered valuable insights but remain...

Time-of-Check Time-of-Use Attacks Against LLMs

This is a nice piece of research: "Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents".: Abstract: Large Language Model LLM-enabled agents are rapidly emerging across a wide range of applications, but their deployment introduces vulnerabilities with security...