When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-Offs

Jailbreak defenses are essential for protecting large language models LLMs, but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cos...

relay-secreport-bench

secreport-bench — the Security-Report Benchmark An evaluation...

cis-hardening-metasploitable2

CIS Benchmark Hardening in Practice: A Full Attack-to-Remediat...

MulRobBench: A Decision-Level Benchmark for Safe and Security-Policy-Compliant Multimodal UAV Agents

Smart-city airspace is transforming Uncrewed Aerial Vehicles UAVs from passive sensing platforms into cyber-physical decision makers that must follow operational rules under degraded observations and ambiguous language. Existing UAV and multimodal benchmarks evaluate perception, navigation,...

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Large Language Models LLMs are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently...

False Prophets: On the Security of World Models in Agentic Systems

Large language models now power autonomous agents capable of complex, multi-step tasks in different environments. Accurate and reliable execution of these tasks requires the agent to predict the results of its actions. Recent research proposes to enhance predictive capabilities via specially...

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra security vulnerabilities, as seemingly harmless skills can contain latent safety risks that only emerge during actual execution. In this work, we...

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models LLMs demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight...

redteam-ai-benchmark — Updated!

Red Team AI Benchmark Russian version: README.ru.md Red Team AI Benchmark is a CLI model-evaluation benchmark. It measures how LLMs understand and respond to red-team questions and security scenarios; it is not a tool for carrying out those activities. Version 2 uses a rubric-based dataset instea...

Trusted Credentials, Untrusted Behavior: Benchmarking LLM-Agent Security in High-Performance Computing

Large language model LLM agents are starting to take on routine work in high-performance computing HPC, including monitoring Slurm jobs, diagnosing failed builds, inspecting simulation output, and coordinating scientific workflows. To do this work, an agent commonly acts under its user's...

Salience Induction against Multi-Hop RAG Agents: Threat and Defense

Agentic retrieval-augmented generation RAG systems increasingly retrieve external evidence and orchestrate tools for knowledge-intensive applications. In Multi-Hop question answering, agents chain facts across documents. Existing defenses focus on content poisoning, which injects false facts, and...

Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

Online scams continue to cause substantial financial and personal harm. As a result, detection systems based on Large Language Models LLMs have been integrated into security products ranging from email gateways and browser extensions to fraud-monitoring dashboards. As this adoption accelerates, a...

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents

Persistent external memory enhances agent continuity but introduces persistent security vulnerabilities: adversarial content can be injected via standard interaction channels, retained across turns, and later distort downstream behavior. To address this challenge, we propose MemPoison, a...

dataset-format-benchmark (0.0.10) potentially affected by CVE-2026-58659 via pytorch-lightning (1.9.2)

pytorch-lightning PYPI version 1.9.2 is affected by a known vulnerability. The following packages have a transitive dependency on pytorch-lightning and may be impacted: - dataset-format-benchmark 0.0.10 Source cves: CVE-2026-58659 Source advisory: OSV:PYSEC-2026-3967...

Traffic-Aware Randomized Smoothing for LLM-Based Network Intrusion Detection

Large language model LLM-based intrusion detection systems IDS are increasingly studied for security monitoring, yet their robustness against feasible traffic manipulation remains largely empirical. We present Traffic-Aware Randomized Smoothing TA-RS, a classifier-agnostic certified defense that...

aider-chat (>=0.82.3, <=0.85.2), aider-code (>=0.1.0.dev19, <=0.1.1.post14) +186 more potentially affected by CVE-2026-59197 via pillow (11.2.1)

pillow PYPI version 11.2.1 is affected by a known vulnerability. The following packages have a transitive dependency on pillow and may be impacted: - aider-chat =0.82.3, =0.1.0.dev19, =0.0.1, =3.1.20260611, =0.1.0, =0.2.0, =0.1.0, =2026.5.28, =2026.4.15, =2.0.6, =25.5.0, =0.0.0, =1.4.3, =0.5.0b2,...

dataset-format-benchmark (0.0.10) potentially affected by CVE-2026-31221 via pytorch-lightning (1.9.2)

pytorch-lightning PYPI version 1.9.2 is affected by a known vulnerability. The following packages have a transitive dependency on pytorch-lightning and may be impacted: - dataset-format-benchmark 0.0.10 Source cves: CVE-2026-31221 Source advisory: OSV:PYSEC-2026-3043...

dataset-format-benchmark (>=0.0.11, <=0.0.12), hybrid-graph (>=0.2.0, <=0.61.0) +2 more potentially affected by CVE-2026-31221 via pytorch-lightning (1.9.3)

pytorch-lightning PYPI version 1.9.3 is affected by a known vulnerability. The following packages have a transitive dependency on pytorch-lightning and may be impacted: - dataset-format-benchmark =0.0.11, =0.2.0, =0.1.0, =0.1.1 Source cves: CVE-2026-31221 Source advisory: OSV:PYSEC-2026-3043...

benchmark-runner (>=1.0.363, <=1.0.441), build-magic (>=0.5.1rc1, <=0.5.1) +3 more potentially affected by CVE-2026-44405 via paramiko (2.10.1)

paramiko PYPI version 2.10.1 is affected by a known vulnerability. The following packages have a transitive dependency on paramiko and may be impacted: - benchmark-runner =1.0.363, =0.5.1rc1, =1.0.0, =1.9.1, =1.9.3 Source cves: CVE-2026-44405 Source advisory: OSV:PYSEC-2026-2858...

appollo (>=1.0.0, <=1.1.1), bdcp (>=1.1.2, <=1.6.8) +14 more potentially affected by CVE-2026-44405 via paramiko (3.0.0)

paramiko PYPI version 3.0.0 is affected by a known vulnerability. The following packages have a transitive dependency on paramiko and may be impacted: - appollo =1.0.0, =1.1.2, =1.0.442, =0.0.16, =0.1.0, =2.6.0, =2.1.1, =2.0.0, =1.3.1, =1.2.0, =0.2.3, =0.2.5 - pydin 0.2.0 and more Source cves:...