terminal-bench-2

Terminal-Bench 2.0 | | | | || || | |/ \ '| ' | | ' \ / | | || || | | / | | | | | | | | | | | | | | || || |||| || || |||| ||,|| |||| || | | | | \ / \ \\ | \ / \ ' \ / | ' \ || | | | \\ | | | / | | | | | | | / / | || | \ \ |/ || |||| || |/ \\ Terminal-Bench is a popular benchmark for...

Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments

Coding agents increasingly run inside organizations whose security controls scoped credentials, restricted egress, read-only filesystems, non-root execution constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is...

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1,968 tasks across five terminal-agent benchmarks and find 323 16% hackable by frontier models given only the task description. This corrupts both...