lava

LAVA: Large Scale Automated Vulnerability Addition Evaluating and improving bug-finding tools is currently difficult due to a shortage of ground truth corpora i.e., software that has known bugs with triggering inputs. LAVA attempts to solve this problem by automatically injecting bugs into...

dynast-bench

DynAST-Bench A DAST benchmark of intentionally-vulnerable apps with ground-truth answer keys for scoring scanners. ⚠️ This repository contains DELIBERATELY INSECURE applications. They exist only to benchmark security tooling - DAST scanners, SAST engines, and LLM security agents. Every app binds t...

ethibench

From Controlled to the Wild: Evaluation of Pentesting Agents in the Real-World AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which systems will perform best on real-world targets. Most existing evaluations...

MAL-2026-16471 Malicious code in z-deno-truth-va499w (npm)

--- -= Per source details. Do not edit below this line.=- Source: amazon-inspector 3f8814711e10416bfe039fa71469c13072ef8d20a1f6f654130ae96ebb38c443 The package was found to contain malicious code or consuming dependency that contains malicious code Source: ghsa-malware...

MAL-2026-16472 Malicious code in z-deno-truth-ya1t4m (npm)

--- -= Per source details. Do not edit below this line.=- Source: amazon-inspector f4ca6bf79ee3672fa881c285bd8af8fc4af19fb3be8e90212de7e8a0be70cf30 The package was found to contain malicious code or consuming dependency that contains malicious code Source: ghsa-malware...

Malicious code in z-deno-truth-va499w (npm)

--- -= Per source details. Do not edit below this line.=- Source: amazon-inspector 3f8814711e10416bfe039fa71469c13072ef8d20a1f6f654130ae96ebb38c443 The package was found to contain malicious code or consuming dependency that contains malicious code Source: ghsa-malware...

Malicious code in z-deno-truth-ya1t4m (npm)

--- -= Per source details. Do not edit below this line.=- Source: amazon-inspector f4ca6bf79ee3672fa881c285bd8af8fc4af19fb3be8e90212de7e8a0be70cf30 The package was found to contain malicious code or consuming dependency that contains malicious code Source: ghsa-malware...

Malicious code in z-deno-truth-bwhlsz (npm)

--- -= Per source details. Do not edit below this line.=- Source: amazon-inspector db0813755b79e157c644502848b1ff5af003fc82fb842b39676490d00f33e814 The package's exports field maps the ./package.json subpath to setup.js, so any consumer resolving require'z-deno-truth-bwhlsz/package.json' executes...

lava v3.3.1

LAVA: Large Scale Automated Vulnerability Addition Evaluating and improving bug-finding tools is currently difficult due to a shortage of ground truth corpora i.e., software that has known bugs with triggering inputs. LAVA attempts to solve this problem by automatically injecting bugs into...

From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code...

MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents

Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets. The challenge is structural: existing safety alignment evaluates overt requests in isolation, leaving models blind to malicious end-states that emerge from...

Spring Office Hours Podcast: S5E14 - Spec Driven Development with Simon Martinelli

Join Dan Vega and DaShaun Carter for the latest updates from the Spring Ecosystem. In this episode, Dan and DaShaun are joined by Java Champion, Vaadin Champion, and Oracle ACE Pro Simon Martinelli to talk about Spec-Driven Development. With AI reshaping how we write code, Simon makes the case th...

A Ground-Truth-Based Evaluation of Vulnerability Detection across Multiple Ecosystems

Automated vulnerability detection tools are widely used to identify security vulnerabilities in software dependencies. However, the evaluation of such tools remains challenging due to the heterogeneous structure of vulnerability data sources, inconsistent identifier schemes, and ambiguities in...

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model LLM agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The...

RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code

How do security scanners perform on real-world code? We present RealVuln, the first open-source benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on 26 intentionally vulnerable Python repositories educational and Capture-The-Flag applications with 796...

SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents

We present SIR-Bench, a benchmark of 794 test cases for evaluating autonomous security incident response agents that distinguishes genuine forensic investigation from alert parroting. Derived from 129 anonymized incident patterns with expert-validated ground truth, SIR-Bench measures not only...

RedShell: A Generative AI-Based Approach to Ethical Hacking

The application of Machine Learning techniques in code generation is now a common practice for most developers. Tools such as ChatGPT from OpenAI leverage the natural language processing capabilities of Large Language Models to generate machine code from natural language descriptions. In the...

A Large-Scale Empirical Study on the Generalizability of Disclosed Java Library Vulnerability Exploits

Open-source software supply chain security relies heavily on assessing affected versions of library vulnerabilities. While prior studies have leveraged exploits for verifying vulnerability affected versions, they point out a key limitation that exploits are version-specific and cannot be directly...

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

System prompt configuration can make the difference between near-total phishing blindness and near-perfect detection in LLM email agents. We present PhishNChips, a study of 11 models under 10 prompt strategies, showing that prompt-model interaction is a first-order security variable: a single...

OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

Synthetic insider threat benchmarks face a consistency problem: corpora generated without an external factual constraint cannot rule out cross-artifact contradictions. The CERT dataset -- the field's canonical benchmark -- is also static, lacks cross-surface correlation scenarios, and predates th...