anamnesis-release

Anamnesis: LLM Exploit Generation Evaluation This repository contains the evaluation framework for studying how LLM agents generate exploits from vulnerability reports in the presence of exploit mitigations. Given a bug report and proof-of-concept trigger, agents analyze vulnerable software and...

DELTA

DELTA: SDN SECURITY EVALUATION FRAMEWORK We now manage DELTA in new repository https://github.com/seungsoo-lee/DELTA. Please visit our new repository and hope you see there again. Thank you! Best Regards, -- DELTA Team...

ethibench

From Controlled to the Wild: Evaluation of Pentesting Agents in the Real-World AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which systems will perform best on real-world targets. Most existing evaluations...

AURA-Eval: Evaluation Framework for Acting under Risk Awareness in LLM Agent Trajectories

LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score, obscuring risk recognition, pre-action detection, and safe task completion when a safe solution exists. We introduce AURA-Eval, a framework combini...

Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systems

Autonomous multi-agent systems AMAS built on large language models LLMs, such as Hermes, increasingly rely on inference-time harnesses to coordinate reasoning and action. Constructing these harnesses requires substantial engineering effort and computational resources, as they are iteratively...

Just Testing, Move Along: Evasion of LLM-Based System Log Interpretation by Prompt Injection

Large Language Models LLMs are increasingly integrated into Security Operations Center SOC workflows, where they support analysts in tasks such as the interpretation of system logs. However, the ability of LLMs to directly process untrusted textual input also introduces new attack surfaces. In...

Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

Online scams continue to cause substantial financial and personal harm. As a result, detection systems based on Large Language Models LLMs have been integrated into security products ranging from email gateways and browser extensions to fraud-monitoring dashboards. As this adoption accelerates, a...

Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept PoC exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework wit...

Turn specs into evals for any agent with ASSERT

Today, we’re releasing Adaptive Spec-driven Scoring for Evaluation and Regression Testing ASSERT, an open-source framework for turning natural-language behavior specifications into executable evaluations. Every team building an AI system starts with a clear intention for the behaviors they want t...

Closing the Sim-To-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR

Leading commercial endpoint detection and response EDR products have shifted from operator-configured rule sets to multi-component systems where autonomous AI components operate alongside, and increasingly in place of, operator-deployed policies. Autonomous defense agents using commercial EDR as...

SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents

Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent...

Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoning

The rapid proliferation of multi-cloud and SaaS platforms has transformed Identity Security Posture Management ISPM into a fundamentally cross-vendor challenge: critical misconfigurations and privilege escalation paths increasingly span multiple identity providers, infrastructure layers, and...

MRMMIA: Membership Inference Attacks on Memory in Chat Agents

Membership inference attacks MIAs test whether a target data record belongs to a system's private data, and have become a standard tool to measure privacy leakage in machine learning systems. Prior work has primarily focused on training corpora or retrieval databases. However, MIAs against agent...

AVISE: Framework for Evaluating the Security of AI Systems

As artificial intelligence AI systems are increasingly deployed across critical domains, their security vulnerabilities pose growing risks of high-profile exploits and consequential system failures. Yet systematic approaches to evaluating AI security remain underdeveloped. In this paper, we...

LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software

Logical vulnerabilities in software stem from flaws in program logic rather than memory safety, which can lead to critical security failures. Although existing automated program repair techniques primarily focus on repairing memory corruption vulnerabilities, they struggle with logical...

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performancee.g., WebArena or safety against malicious actionse.g., SafeArena, no existing framework assesses an agent's ability to...

Why Aggregate Accuracy Is Inadequate for Evaluating Fairness in Law Enforcement Facial Recognition Systems

Facial recognition systems are increasingly deployed in law enforcement and security contexts, where algorithmic decisions can carry significant societal consequences. Despite high reported accuracy, growing evidence demonstrates that such systems often exhibit uneven performance across demograph...

Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-Agent AI Systems

As Large Language Models LLMs and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to...

From Storage to Steering: Memory Control Flow Attacks on LLM Agents

Modern agentic systems allow Large Language Model LLM agents to tackle complex tasks through extensive tool usage, forming structured control flows of tool selection and execution. Existing security analyses often treat these control flows as ephemeral, one-off sessions, overlooking the persisten...

New RFP Template for AI Usage Control and AI Governance 

As AI becomes the central engine for enterprise productivity, security leaders are finally getting the green light — and the budget — to secure it. But there’s a quiet crisis unfolding in the boardroom: many organizations know they need "AI Governance," but they have no idea what they are actuall...