CTFTiny

CTFTiny: Lite Benchmarking Offensive Cyber Skills in Large Language Models This is the official repository for CTFTiny from "Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark" AAAI'26 paper. For CTFJudge, please refer to CTFJud...

promptfoo

Promptfoo: evaluaciones de LLM y red teaming promptfoo es una CLI y biblioteca para evaluar y hacer red teaming de aplicaciones LLM. Abandona el enfoque de prueba y error: comienza a lanzar aplicaciones de IA seguras y confiables. Sitio web · Primeros pasos · Red Teaming · Documentación ·...

reverse-captcha-eval

Reverse CAPTCHA: Evaluación de la susceptibilidad de los LLM a la inyección invisible de instrucciones Unicode Un marco de evaluación que prueba si los modelos de lenguaje grandes siguen instrucciones codificadas en Unicode invisible incrustadas en texto que de otro modo parece normal. Mientras q...

bloom

Bloom: Automated Behavioral Evaluations for LLMs !IMPORTANT Bloom has a new home. It is now developed and maintained by Meridian Labs and lives at meridianlabs-ai.github.io/petribloom — all new features and fixes will land there. This repository is frozen at its last standalone release and will n...

MalEval

MalEval Article: Is “Knowing It’s Malicious” Enough? Evaluating LLMs for Fine-Grained Malware Behavior Auditing Article DOI: 10.1145/3832187 MalEval is a framework for evaluating Android malware behavior reports generated by large language models. The code in this repository implements two...

claude_opus_cve_2023_0266

Demonstration that Claude 3 Opus does not understand CVE-2023-0266 and does not find it Demo 1. Even if told where the bug is Opus does not find it, and hallucinates the presence of lock acquisitions Demo 2. "Prompt engineering" aka telling the LLM exactly how to find the bug also doesn't work De...

redteam-ai-benchmark

Red Team AI Benchmark Versión en ruso: README.ru.md Red Team AI Benchmark es un benchmark de evaluación de modelos CLI. Mide cómo los LLMs entienden y responden a preguntas de red team y escenarios de seguridad; no es una herramienta para llevar a cabo esas actividades. La versión 2 utiliza un...

promptfoo v0.122.1

Promptfoo: LLM evals & red teaming promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps. Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI. Promptfoo...

Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection

Large Language Models LLMs have demonstrated significant potential in automated software security, particularly in vulnerability detection. However, existing benchmarks primarily focus on isolated, single-vulnerability samples or function-level classification, failing to reflect the complexity of...

LLM-GUARD: Large Language Model-Based Detection and Repair of Bugs and Security Vulnerabilities in C++ and Python

Large Language Models LLMs such as ChatGPT-4, Claude 3, and LLaMA 4 are increasingly embedded in software/application development, supporting tasks from code generation to debugging. Yet, their real-world effectiveness in detecting diverse software bugs, particularly complex, security-relevant...