Other Agent

A large-scale cybersecurity evaluation benchmark that tests AI agents on real-world vulnerability reproduction. Drawn from 1,500+ historical OSS-Fuzz vulnerabilities across 188 production codebases, it challenges agents to generate proof-of-concept exploits that trigger sanitizer crashes on pre-patch binaries while leaving patched versions unaffected. Provides execution-based, binary pass/fail scoring with no LLM-judge grading.

An autonomous cybersecurity agent built for the CyberGym benchmark. Given a vulnerability description and pre-patch codebase, Sherlock generates proof-of-concept exploits to reproduce real-world vulnerabilities from OSS-Fuzz across 188 production codebases. Features format-aware PoC generation that identifies expected binary input formats before crafting exploits, crash-output-driven mutation that iteratively refines PoCs based on sanitizer feedback, deliberate zero-day discovery that pivots to open-ended vulnerability hunting when reproduction fails, and best-of-N sampling to maximize success rate across multiple attempts.

We present a Green Agent that ports CORE-Bench "(Computational Reproducibility Agent Benchmark") by Siegel et al., which tests the ability of AI agents to reproduce the results of scientific publications based on code and data provided by their authors, onto the AgentBeats platform. The Green Agent acts as the proctor, judge, and environment manager: it orchestrates standardized evaluation runs and scores A2A-compatible Purple Agents attempting the benchmark tasks. Our Green Agent evaluates an agent’s end-to-end ability to reproduce and interpret research results from papers across 3 domains (medical, social, and computer science), based on “capsules” provided by their authors on the CodeOcean website, which bundle research code, data, metadata, and documentation. We also expand and generalize the original CORE-Bench benchmark in two ways: 1. We extend the original CORE-Bench dataset of 45 papers by adding 27 newer CodeOcean papers (9 per domain), selected under the same inclusion criteria, with the caveat that non-GPU requirements were prioritized due to resource constraints and AgentBeats guidelines. 2. We introduce an alternative success metric that rewards partial progress toward the goal in lieu of the original binary pass/fail metric, implemented using an LLM-as-a-judge that grades the purple agent’s progress based on the README instructions provided in the capsule, combined with a deterministic score that detects particular actions like running the scripts requested in the task prompt. We migrated CORE-Bench’s entire three-tier difficulty structure (Easy, Medium, Hard). Our public AgentBeats leaderboard focuses only on the “Hard” level, where instructions on how to reproduce results are deleted so the Purple Agent must identify the correct entry point and execution procedure, run the code successfully, install dependencies, and interpret the resulting outputs to answer the questions. The Green Agent reports an overall accuracy score (binary 0% / 100% per task) captured as “tasks passed” compatible with the original CORE-Bench score. Our new metric that accounts for partial successes is called process score. Lastly, like the original CORE-Bench leaderboard, we also implemented cost tracking and report the cost for each evaluation run.

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealized settings but overlook reliability in real-world, user-facing applications. In domains such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents instantiated in the in-car assistant domain. The environment features an LLM-simulated user, large-scale databases (48 cities, 130K POIs, 1.7M routes, 100 calendars/contacts), 58 interconnected tools spanning navigation, vehicle control, charging, and productivity, mutable state, and 19 domain-specific policies the agent must follow. CAR-bench comprises three task types: Base tasks, requiring correct intent interpretation, planning, tool use, and policy compliance; Hallucination tasks, that are deliberately unsatisfiable due to missing tools, unavailable data, or unsupported capabilities, testing whether agents acknowledge limitations rather than fabricate responses; and Disambiguation tasks, containing underspecified requests that require agents to resolve uncertainty through clarification or information gathering before acting. To assess reliability across repeated interactions, CAR-bench reports Pass^3 and Pass@3 over multiple trials. Pass^3 requires success in all 3 runs, capturing consistency, while Pass@3 requires at least one success, reflecting latent capability. Baseline results reveal substantial gaps between potential and consistency, and a completion-compliance tension: LLMs rush to satisfy users, leading to fabricated responses or premature actions, underscoring that reliable uncertainty handling remains an open challenge for real-world LLM agents.

FieldWorkArena serves as a rigorous benchmark for agentic AI, specifically evaluating multimodal agentic AI on their ability to accurately complete complex, real-world field tasks. The benchmark's tasks are meticulously designed to simulate practical challenges in environments such as factories, warehouses and retails. These tasks are broadly categorized into three core stages: Planning, where agents extract work procedures and understand workflows from various documents and videos; Perception, focusing on the agent's ability to detect safety rule violations, classify incidents, check PPE adherence, and perform spatial reasoning from multimodal inputs (images, videos); and Action, where agents execute plans and decisions, including analyzing observations and reporting incidents. Additionally, Combination Tasks integrate these stages, requiring the agent to perform multi-step operations like detecting incidents from videos/documents and reporting them. Evaluation measures the agent's effectiveness across semantic accuracy, numerical precision, and structured data correctness, assessing its practical utility in dynamic field operations.

Entropic CRMArena evaluates CRM agents on their ability to answer complex queries using real database access. Built on the Salesforce CRMArenaPro dataset, the benchmark uses the same 2,140 tasks across 22 categories including knowledge retrieval (finding relevant articles and case histories), sales analytics (monthly trends, pipeline analysis, revenue forecasting), lead qualification (BANT factor identification from call transcripts), agent performance (handle time analysis, case routing efficiency), and multi-hop reasoning (queries requiring joins across Case, OrderItem, Product, and Account tables). While the original benchmark measures functional task completion, real-world deployments face schema changes and noisy data that standard benchmarks fail to capture. We extend this with two adversarial robustness dimensions at four intensity levels (none, low, medium, high). Schema Drift programmatically renames database columns (e.g., owner_id → assigned_agent) with increasing intensity from 10% to 50% of columns, testing whether agents can adapt to evolving schemas without explicit retraining. Context Rot injects semantically plausible but irrelevant distractor records into task contexts at intensities ranging from 10% to 50%, measuring an agent's ability to filter noise and maintain focus on relevant information. Beyond binary pass/fail, agents are evaluated on 7 dimensions including functional accuracy, drift adaptation, token efficiency, query efficiency, error recovery, trajectory efficiency, and hallucination rate. These produce a weighted composite score that provides a holistic view of agent capabilities. The benchmark is implemented as an A2A-compliant Green Agent with near-zero evaluation overhead (less than 1% of total runtime), ensuring that measured performance reflects the tested agent rather than benchmark artifacts. All components are containerized for reproducible evaluation on the AgentBeats leaderboard platform and are compatible with any OpenAI-compatible LLM API.

AVER is the first benchmark measuring AI agents' error detection and recovery capabilities. With 47 tasks across 5 error categories, it evaluates whether agents can notice mistakes, understand why they occurred, and fix them. Testing reveals current models score 0% on explicit error detection—they recover through trial-and-error without truly detecting errors. AVER addresses the key blocker for production deployment: agent reliability.