Cybersecurity Agent

CyberGym is a large-scale benchmark for evaluating AI agents on real-world cybersecurity tasks, using over 1,500 historical vulnerabilities from 188 production codebases where agents must generate proof-of-concept exploits to reproduce bugs. It emphasizes realistic, execution-based evaluation and demonstrates both the difficulty of vulnerability analysis and agents’ emerging ability to discover new security flaws.

Ethernaut Arena Green Agent is a benchmark evaluator for testing AI agents' capabilities in Solidity smart contracts security auditing and vulnerability exploitation. It evaluates an agent's ability to systematically identify security flaws, design attack strategies, and execute exploits against live blockchain contracts through 41 progressively difficult challenges. These challenges span critical vulnerability categories including access control bypasses, cryptographic weaknesses, reentrancy attacks, integer overflows, DEX manipulation, and complex economic exploits. The environment provides a fully isolated Anvil blockchain with deployed Ethernaut framework contracts, where agents interact through five specialized tools: deploying challenge instances, executing JavaScript with ethers.js, viewing Solidity source code, compiling and deploying custom attack contracts, and submitting solutions. Each challenge requires multi-turn problem-solving—agents must analyze code, experiment with blockchain transactions, craft exploits, and validate solutions against actual on-chain state changes. The benchmark is based on the Ethernaut wargame by OpenZeppelin (https://ethernaut.openzeppelin.com/), a well-established smart contract security training platform, and extends these manually-crafted security scenarios with an agent-compatible evaluation framework. Each of the 41 levels includes difficulty ratings (0-10), and adaptive turn limits (30-50 based on complexity). Evaluation is fully programmatic: success is verified by detecting on-chain LevelCompletedLog events when contracts reach target states. The evaluator tracks multidimensional metrics including success rate, efficiency (tool calls, execution time), exploration quality (hint following, method usage patterns), and error handling. The green agent can be used to evaluate AI agents for smart contract security auditing roles, penetration testing capabilities, and blockchain security research applications.

The RCA-Bench green agent evaluates an agent’s ability to perform root-cause analysis of security vulnerabilities in real-world codebases. It leverages the ARVO dataset to retrieve programs with known bugs discovered through fuzzing. For each task, the green agent prepares a realistic debugging scenario and provides the corresponding codebase to the purple agent. The purple agent is then evaluated on its ability to identify the root cause of the vulnerability by localizing the relevant files and lines of code. This benchmark tests an agent’s capacity to reason over large codebases and accurately pinpoint the source of security-critical bugs.

Title: Cyber Security Evaluator: MITRE-Aligned Adaptive Security Benchmarking Abstract: The Cyber Security Evaluator is a Green Agent that identifies and evaluates specific MITRE ATT&CK techniques to benchmark "Purple Agent" security detectors. It employs an adaptive 7-agent ecosystem—including Thompson Sampling for testing strategy and Novelty Search for evasion discovery—to generate evolving attack campaigns. Focusing on techniques like SQL Injection and Prompt Injection (LLM Jailbreaks), evaluations are conducted within a secure Docker sandbox. The agent provides distinct MITRE coverage mapping and performance metrics, helping developers validate their angebts against recognized adversary behaviors and real-world threats.

Skylark's Lumi Scout is a bootstrapped cybersecurity bot designed to solve cybersecurity-related challenges in an as efficient method as possible.

CyberGym Green Agent: AI-Powered Vulnerability Exploitation Assessment Our green agent evaluates AI agents (purple agents) on their ability to discover and exploit real-world software vulnerabilities from the OSS-Fuzz dataset. Tasks: - Purple agents receive vulnerability task IDs (e.g., oss-fuzz:42535201) - They must generate Proof-of-Concept (PoC) binary exploits - The green agent validates PoCs against vulnerable binaries using differential testing Key Features: 1. A2A Protocol Integration: Full compliance with AgentBeats message/send JSON-RPC 2. CyberGym Benchmark: Leverages UC Berkeley's CyberGym dataset with real vulnerabilities from projects like OpenSSL, FFmpeg, and libmspack 3. Surgical Data Bundling: Optimized Docker image (2GB) containing vulnerability binaries for efficient CI/CD execution 4. Mock Validation Fallback: Transparent Phase 1 validation for pipeline integrity demonstration Scoring: - Pass rate based on successful PoC generation - 100 points per task for valid exploits - Transparent reporting of validation mode This green agent establishes the foundation for evaluating AI agents' capabilities in automated vulnerability discovery and exploitation - a critical skill for next-generation cybersecurity tools.

WaspWatch evaluates web agents against prompt injection attacks using the official Meta FAIR WASP benchmark. Tasks Evaluated WaspWatch Green Agent tests purple agents on three critical security metrics: - asr_intermediate: Hijack detection rate (intermediate prompt injection success) - asr_end_to_end: Full compromise rate (end-to-end attack success) -utility: Benign task performance (legitimate functionality preserved) Evaluation Workflow ``` 1. Purple agent Docker image → /assess endpoint 2. WASP benchmark (VisualWebArena) → GitLab/Reddit tasks 3. Automated attacks → Prompt injections 4. Metrics extraction → JSON results 5. Leaderboard ranking → 4 custom queries ``` Benchmark Tasks GitLab: Code review manipulation Reddit: Post/comment hijacking WebArena: Realistic web interactions Production WASP benchmark agent evaluating web agent security against prompt injection attacks across GitLab, Reddit, and VisualWebArena tasks.

Avayam is a research-grade cybersecurity benchmark that evaluates AI agents on their ability to remediate real-world vulnerabilities. It agentifies the MSR 2020 dataset (Fan et al.), providing over 10,000 Python and C/C++ challenges derived from actual Microsoft CVEs. Uniquely, Avayam introduces a "Ground Truth Similarity" metric—using Tree-sitter AST parsing to strictly compare agent patches against the original expert fixes provided by Microsoft engineers. This ensures that agents are scored not just on passing tests, but on adhering to secure coding standards and reproducing canonical security patches