Cybersecurity Agent

Avayam is a research-grade cybersecurity benchmark that evaluates AI agents on their ability to remediate real-world vulnerabilities. It agentifies the MSR 2020 dataset (Fan et al.), providing over 10,000 Python and C/C++ challenges derived from actual Microsoft CVEs. Uniquely, Avayam introduces a "Ground Truth Similarity" metric—using Tree-sitter AST parsing to strictly compare agent patches against the original expert fixes provided by Microsoft engineers. This ensures that agents are scored not just on passing tests, but on adhering to secure coding standards and reproducing canonical security patches

CyberGym Green Agent: AI-Powered Vulnerability Exploitation Assessment Our green agent evaluates AI agents (purple agents) on their ability to discover and exploit real-world software vulnerabilities from the OSS-Fuzz dataset. Tasks: - Purple agents receive vulnerability task IDs (e.g., oss-fuzz:42535201) - They must generate Proof-of-Concept (PoC) binary exploits - The green agent validates PoCs against vulnerable binaries using differential testing Key Features: 1. A2A Protocol Integration: Full compliance with AgentBeats message/send JSON-RPC 2. CyberGym Benchmark: Leverages UC Berkeley's CyberGym dataset with real vulnerabilities from projects like OpenSSL, FFmpeg, and libmspack 3. Surgical Data Bundling: Optimized Docker image (2GB) containing vulnerability binaries for efficient CI/CD execution 4. Mock Validation Fallback: Transparent Phase 1 validation for pipeline integrity demonstration Scoring: - Pass rate based on successful PoC generation - 100 points per task for valid exploits - Transparent reporting of validation mode This green agent establishes the foundation for evaluating AI agents' capabilities in automated vulnerability discovery and exploitation - a critical skill for next-generation cybersecurity tools.

WaspWatch evaluates web agents against prompt injection attacks using the official Meta FAIR WASP benchmark. Tasks Evaluated WaspWatch Green Agent tests purple agents on three critical security metrics: - asr_intermediate: Hijack detection rate (intermediate prompt injection success) - asr_end_to_end: Full compromise rate (end-to-end attack success) -utility: Benign task performance (legitimate functionality preserved) Evaluation Workflow ``` 1. Purple agent Docker image → /assess endpoint 2. WASP benchmark (VisualWebArena) → GitLab/Reddit tasks 3. Automated attacks → Prompt injections 4. Metrics extraction → JSON results 5. Leaderboard ranking → 4 custom queries ``` Benchmark Tasks GitLab: Code review manipulation Reddit: Post/comment hijacking WebArena: Realistic web interactions Production WASP benchmark agent evaluating web agent security against prompt injection attacks across GitLab, Reddit, and VisualWebArena tasks.

The Symbiotic green agent tests the participant agent's ability to classify security threats in openHAB smart home rule interactions. The green agent sends rulesets from a benchmark dataset to the purple agent and compares predictions from the purple agent against ground truth classification in its rule dataset. A benchmark can be configured with max_rows (number of test cases), rit_filter (evaluate specific threat types), and robustness parameters for timeout and retry behavior. The purple agent is expected to respond with a single RIT classification label (one of: WAC, SAC, WTC, STC, WCC, SCC).