promptfoo

Promptfoo: LLM evals & red teaming promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. Stop using trial-and-error... start shipping secure, reliable agents Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI. Promptfoo remain...

bordair-detector

Bordair Detector A two-stage detector for prompt injection and jailbreak attempts in LLM inputs. A regex gate settles the easy majority of traffic without touching the model; anything ambiguous falls through to a quantised DeBERTa-v3 classifier running in ONNX. The same engine covers image,...

llm-security-101

LLM Security 101 Delving into the Realm of LLM Security: An Exploration of Offensive and Defensive Tools, Unveiling Their Present Capabilities. As we embrace Large Language Models LLMs in various applications and functionalities, it is crucial to grasp the associated risks and actively mitigate, ...

cover

Cover Keep private data, internal infrastructure and secrets out of cloud coding agents without breaking your workflow. Install · Quick start · Policies · Monitoring · Pi / OMP · Security Cover is a bidirectional privacy proxy for AI coding agents. It replaces matched sensitive values locally wit...

OpenHunterAI

OpenHunterAI Your local AI red team. Attacker-style reasoning for web, API, and LLM application security. Quick start · Agent skill · Assessment model · Architecture · Docs · Star history OpenHunterAI brings scope, scan activity, findings, and remediation into one local workspace. Start without a...

LLM-MCP-Security-Field-Guide

🛡️ AI Security Field Guide — LLM + MCP Security The most comprehensive, up-to-date, practitioner-first security reference for LLM applications and Model Context Protocol MCP deployments. Covers real CVEs, live attack patterns, OWASP frameworks, red teaming tools, and actionable checklists — update...

DonkAI

Hands-on lab for the OWASP Top 10 for LLM Applications 2025 - no real LLM required. DonkAI is deliberately vulnerable web app you can run in one command and use to learn how LLM-integrated systems get broken by actually breaking them. Every OWASP LLM Top 10 category is represented by at least one...

reasongate

ReasonGate A self-hostable gate that inspects the text going into and out of an LLM and returns an explainable allow / flag / block decision with a machine-readable audit record for every call. What this is The open-source core is rule-based. It does four things: recognizes known prompt-injection...

SecureAI-Scan

SecureAI-Scan An offline scanner for the code that talks to LLMs, MCP servers, vector stores, and Agent Skills. It finds prompt injection, MCP command injection and path traversal, RAG data leaks, and poisoned skills, and it proves each finding with a source → sink trace through your real imports...

llm-prompt-injection-resources

llm-prompt-injection-resources A curated collection of resources for learning and researching LLM prompt injection attacks, defenses, and security. Donate Support the maintenance of this project with PayPal or by scanning the QR code below...

aco-prompt-shield

aco-prompt-shield 🛡️ Stop prompt injection attacks before they reach your LLM — zero API costs, runs entirely locally, integrates in 2 minutes. Prompt injection is the 1 security risk for LLM applications. aco-prompt-shield catches known jailbreak patterns, understands semantic intent via ML, and...

ACEA: An Adversarial Co-Evolution Arena for Head-To-Head Red-Team and Blue-Team LLM Testing

Automated red-team attacks and blue-team defenses for large language models LLMs are advancing quickly. However, attackers and defenders are built and tested in isolation, and the resulting scores are hard to trust. To tackle this, we present ACEA Adversarial Co-Evolution Arena, a platform that...

Shifting from Injection to Interaction: Rethinking Web Security in the Age of LLMs and Beyond

Large language models LLMs are becoming integral to web applications and browser agents, transforming online interactions while introducing new attack vectors and reshaping longstanding web vulnerabilities. Classical threats such as cross-site scripting XSS can be amplified through LLM-mediated...

Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-As-Code

Large language models are increasingly used to author Infrastructure-as-Code IaC, where a single insecure default can be deployed directly into production. Prior evaluations report raw vulnerability counts for model-generated IaC, but without a human baseline they cannot determine whether models...

“Sorry, I can’t help with that”: How your guardrails might become the attacker’s best friend

Welcome to this week's edition of the Threat Source newsletter. Hello, everyone. Long time reader, first time writer here at the Threat Source newsletter! I wanted to start out by introducing myself. My colleague and friend Mick Baccio set the bar pretty high last week, so I was planning to tell...

Enhancing User Resilience against AI-Augmented Phishing: A Two-Stage Framework for Detection and Personalized Training

The rapid development of artificial intelligence, including agents and deepfake techniques, has accelerated phishing attacks and lowered the threshold for attackers. Modern phishing attacks now blend multiple tactics, including social engineering, URL spoofing, and AI deepfakes enabling adversari...

Benchmarking Identity-Sensitive LLM Outputs for Surveillance and Security Robots

Large language models LLMs are increasingly used to generate textual robot design specifications, interaction policies, and risk assessments during early-stage robot development. Such outputs may influence how surveillance and security robots are conceptualized, documented, and ultimately...

Breaking and Defending LLM-Powered Social Media Bot Detection Systems

The rise of social media bots poses a persistent threat, enabling misinformation, opinion manipulation, and the erosion of trust in online platforms. To combat this, machine learning systems have been developed to detect and limit bot activity, but attackers continuously adapt through techniques...

Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools

Rapid adoption of large language models LLMs in enterprise settings has introduced operational, security, and governance risks. As generative AI applications move from pilot to production, manual harm identification and mitigation are becoming difficult to scale. Although many tools support model...

CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training

Despite recent advances, frontier large language model LLM agents remain limited in discovering and patching complex vulnerabilities in real-world software. Generally available agents can already aid attackers, who only need to find one exploitable weakness, while defenders must continuously...