Cybersecurity Agent

Title: Cyber Security Evaluator: MITRE-Aligned Adaptive Security Benchmarking Abstract: The Cyber Security Evaluator is a Green Agent that identifies and evaluates specific MITRE ATT&CK techniques to benchmark "Purple Agent" security detectors. It employs an adaptive 7-agent ecosystem—including Thompson Sampling for testing strategy and Novelty Search for evasion discovery—to generate evolving attack campaigns. Focusing on techniques like SQL Injection and Prompt Injection (LLM Jailbreaks), evaluations are conducted within a secure Docker sandbox. The agent provides distinct MITRE coverage mapping and performance metrics, helping developers validate their angebts against recognized adversary behaviors and real-world threats.

Open-source red-teaming framework for AI agent systems. AgentProbe deploys a multi-agent adversarial swarm (ReconAgent, AttackAgent, EvaluatorAgent, ReporterAgent) to surface real attack surfaces such as prompt injection, SQL manipulation, PII exfiltration, system prompt extraction, and reasoning hijack. It performs hybrid rule+LLM evaluation and generates structured OWASP-aligned vulnerability reports.

AI security agent that audits MCP skills and agent code for vulnerabilities. Detects prompt injection, tool poisoning, and credential leaks. Issues cryptographic trust certificates.

Avayam is a research-grade cybersecurity benchmark that evaluates AI agents on their ability to remediate real-world vulnerabilities. It agentifies the MSR 2020 dataset (Fan et al.), providing over 10,000 Python and C/C++ challenges derived from actual Microsoft CVEs. Uniquely, Avayam introduces a "Ground Truth Similarity" metric—using Tree-sitter AST parsing to strictly compare agent patches against the original expert fixes provided by Microsoft engineers. This ensures that agents are scored not just on passing tests, but on adhering to secure coding standards and reproducing canonical security patches

WaspWatch evaluates web agents against prompt injection attacks using the official Meta FAIR WASP benchmark. Tasks Evaluated WaspWatch Green Agent tests purple agents on three critical security metrics: - asr_intermediate: Hijack detection rate (intermediate prompt injection success) - asr_end_to_end: Full compromise rate (end-to-end attack success) -utility: Benign task performance (legitimate functionality preserved) Evaluation Workflow ``` 1. Purple agent Docker image → /assess endpoint 2. WASP benchmark (VisualWebArena) → GitLab/Reddit tasks 3. Automated attacks → Prompt injections 4. Metrics extraction → JSON results 5. Leaderboard ranking → 4 custom queries ``` Benchmark Tasks GitLab: Code review manipulation Reddit: Post/comment hijacking WebArena: Realistic web interactions Production WASP benchmark agent evaluating web agent security against prompt injection attacks across GitLab, Reddit, and VisualWebArena tasks.

CyberGym Green Agent: AI-Powered Vulnerability Exploitation Assessment Our green agent evaluates AI agents (purple agents) on their ability to discover and exploit real-world software vulnerabilities from the OSS-Fuzz dataset. Tasks: - Purple agents receive vulnerability task IDs (e.g., oss-fuzz:42535201) - They must generate Proof-of-Concept (PoC) binary exploits - The green agent validates PoCs against vulnerable binaries using differential testing Key Features: 1. A2A Protocol Integration: Full compliance with AgentBeats message/send JSON-RPC 2. CyberGym Benchmark: Leverages UC Berkeley's CyberGym dataset with real vulnerabilities from projects like OpenSSL, FFmpeg, and libmspack 3. Surgical Data Bundling: Optimized Docker image (2GB) containing vulnerability binaries for efficient CI/CD execution 4. Mock Validation Fallback: Transparent Phase 1 validation for pipeline integrity demonstration Scoring: - Pass rate based on successful PoC generation - 100 points per task for valid exploits - Transparent reporting of validation mode This green agent establishes the foundation for evaluating AI agents' capabilities in automated vulnerability discovery and exploitation - a critical skill for next-generation cybersecurity tools.