Cybersecurity Agent

We introduce BRACEGreen, an IT security pentesting benchmark designed to evaluate agentic pentesting capabilities. The benchmark comprises seven challenges based on VulnHub Capture-The-Flag (CTF) scenarios. Each challenge requires obtaining root privileges on a vulnerable system to retrieve a hidden flag. Unlike traditional CTF evaluations, BRACEGreen enables incremental, offline assessment without requiring actual virtual machines. Each challenge is decomposed into a sequence of mandatory milestones. After each step, the agent receives gold-standard commands and outputs from previous steps and must provide the subsequent command to progress. Evaluation employs an LLM-as-a-judge approach to compare agent-generated commands against pre-defined alternatives. The final score represents the ratio of completed steps to total required steps. Gold solutions were derived from community walkthroughs and enriched with semantically equivalent alternatives using LLM guidance, including identification of dead-end paths. All solutions were rigorously validated by security experts to ensure command-line equivalents accurately complete each CTF challenge on their respective machine.

This green agent evaluates AI agents on real-world vulnerability analysis using the CyberGym benchmark. Given vulnerable source code and a vulnerability description, agents must (1) identify the root cause, (2) generate a proof-of-concept (PoC) input that triggers the vulnerability, and (3) explain their analysis. Scoring combines automated PoC validation via CyberGym's sandboxed execution environment (50 points) with LLM-as-judge evaluation of explanation quality across four dimensions: vulnerability identification, root cause analysis, exploitation path, and fix understanding (50 points). The benchmark tests genuine security reasoning capabilities, not pattern matching, by requiring agents to understand code semantics, craft precise exploit inputs, and articulate their findings. Tasks span real CVEs from the ARVO and OSS-Fuzz datasets with configurable difficulty levels (level0-level3) that progressively reveal more context.