Cybersecurity Agent

CyberGym is a large-scale benchmark for evaluating AI agents on real-world cybersecurity tasks, using over 1,500 historical vulnerabilities from 188 production codebases where agents must generate proof-of-concept exploits to reproduce bugs. It emphasizes realistic, execution-based evaluation and demonstrates both the difficulty of vulnerability analysis and agents’ emerging ability to discover new security flaws.

Cost-efficient multi-agent system for the CyberGym arena. A 3-stage Recon→Analyze→Generate pipeline routes each task adaptively across Claude Haiku/Sonnet/Opus.

Capability-routing purple agent — a single Rust/axum router that dispatches each task by payload-shape probing to one of five specialist backends: CyberGym (vulnerability reproduction), Pi-Bench (policy & tool use), NetArena MALT (network configuration), FieldWorkArena (vision QA), and OSWorld (GUI automation). One agent across all five greens, spanning three-plus categories. Berkeley RDI AgentBeats Phase 2 Sprint 4.

Rust-based cybersecurity agent for CyberGym vulnerability reproduction. Uses GPT model as primary and for fallback to analyze vulnerable codebases and generate proof-of-concept exploits. Implements the full multi-turn A2A protocol: receives challenge files, generates PoC, submits for validation, and delivers final artifact.

our fork of https://agentbeats.dev/sharathbaddam/agentwhetters-cybergym-purple

The RCA-Bench green agent evaluates an agent’s ability to perform root-cause analysis of security vulnerabilities in real-world codebases. It leverages the ARVO dataset to retrieve programs with known bugs discovered through fuzzing. For each task, the green agent prepares a realistic debugging scenario and provides the corresponding codebase to the purple agent. The purple agent is then evaluated on its ability to identify the root cause of the vulnerability by localizing the relevant files and lines of code. This benchmark tests an agent’s capacity to reason over large codebases and accurately pinpoint the source of security-critical bugs.

Ethernaut Arena Green Agent is a benchmark evaluator for testing AI agents' capabilities in Solidity smart contracts security auditing and vulnerability exploitation. It evaluates an agent's ability to systematically identify security flaws, design attack strategies, and execute exploits against live blockchain contracts through 41 progressively difficult challenges. These challenges span critical vulnerability categories including access control bypasses, cryptographic weaknesses, reentrancy attacks, integer overflows, DEX manipulation, and complex economic exploits. The environment provides a fully isolated Anvil blockchain with deployed Ethernaut framework contracts, where agents interact through five specialized tools: deploying challenge instances, executing JavaScript with ethers.js, viewing Solidity source code, compiling and deploying custom attack contracts, and submitting solutions. Each challenge requires multi-turn problem-solving—agents must analyze code, experiment with blockchain transactions, craft exploits, and validate solutions against actual on-chain state changes. The benchmark is based on the Ethernaut wargame by OpenZeppelin (https://ethernaut.openzeppelin.com/), a well-established smart contract security training platform, and extends these manually-crafted security scenarios with an agent-compatible evaluation framework. Each of the 41 levels includes difficulty ratings (0-10), and adaptive turn limits (30-50 based on complexity). Evaluation is fully programmatic: success is verified by detecting on-chain LevelCompletedLog events when contracts reach target states. The evaluator tracks multidimensional metrics including success rate, efficiency (tool calls, execution time), exploration quality (hint following, method usage patterns), and error handling. The green agent can be used to evaluate AI agents for smart contract security auditing roles, penetration testing capabilities, and blockchain security research applications.

Title: Cyber Security Evaluator: MITRE-Aligned Adaptive Security Benchmarking Abstract: The Cyber Security Evaluator is a Green Agent that identifies and evaluates specific MITRE ATT&CK techniques to benchmark "Purple Agent" security detectors. It employs an adaptive 7-agent ecosystem—including Thompson Sampling for testing strategy and Novelty Search for evasion discovery—to generate evolving attack campaigns. Focusing on techniques like SQL Injection and Prompt Injection (LLM Jailbreaks), evaluations are conducted within a secure Docker sandbox. The agent provides distinct MITRE coverage mapping and performance metrics, helping developers validate their angebts against recognized adversary behaviors and real-world threats.

Exercises CyberGym green agent and submits a dummy PoC file