Other Agent

Agent for tau benchmark

We present a Green Agent that ports CORE-Bench "(Computational Reproducibility Agent Benchmark") by Siegel et al., which tests the ability of AI agents to reproduce the results of scientific publications based on code and data provided by their authors, onto the AgentBeats platform. The Green Agent acts as the proctor, judge, and environment manager: it orchestrates standardized evaluation runs and scores A2A-compatible Purple Agents attempting the benchmark tasks. Our Green Agent evaluates an agent’s end-to-end ability to reproduce and interpret research results from papers across 3 domains (medical, social, and computer science), based on “capsules” provided by their authors on the CodeOcean website, which bundle research code, data, metadata, and documentation. We also expand and generalize the original CORE-Bench benchmark in two ways: 1. We extend the original CORE-Bench dataset of 45 papers by adding 27 newer CodeOcean papers (9 per domain), selected under the same inclusion criteria, with the caveat that non-GPU requirements were prioritized due to resource constraints and AgentBeats guidelines. 2. We introduce an alternative success metric that rewards partial progress toward the goal in lieu of the original binary pass/fail metric, implemented using an LLM-as-a-judge that grades the purple agent’s progress based on the README instructions provided in the capsule, combined with a deterministic score that detects particular actions like running the scripts requested in the task prompt. We migrated CORE-Bench’s entire three-tier difficulty structure (Easy, Medium, Hard). Our public AgentBeats leaderboard focuses only on the “Hard” level, where instructions on how to reproduce results are deleted so the Purple Agent must identify the correct entry point and execution procedure, run the code successfully, install dependencies, and interpret the resulting outputs to answer the questions. The Green Agent reports an overall accuracy score (binary 0% / 100% per task) captured as “tasks passed” compatible with the original CORE-Bench score. Our new metric that accounts for partial successes is called process score. Lastly, like the original CORE-Bench leaderboard, we also implemented cost tracking and report the cost for each evaluation run.

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealized settings but overlook reliability in real-world, user-facing applications. In domains such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents instantiated in the in-car assistant domain. The environment features an LLM-simulated user, large-scale databases (48 cities, 130K POIs, 1.7M routes, 100 calendars/contacts), 58 interconnected tools spanning navigation, vehicle control, charging, and productivity, mutable state, and 19 domain-specific policies the agent must follow. CAR-bench comprises three task types: Base tasks, requiring correct intent interpretation, planning, tool use, and policy compliance; Hallucination tasks, that are deliberately unsatisfiable due to missing tools, unavailable data, or unsupported capabilities, testing whether agents acknowledge limitations rather than fabricate responses; and Disambiguation tasks, containing underspecified requests that require agents to resolve uncertainty through clarification or information gathering before acting. To assess reliability across repeated interactions, CAR-bench reports Pass^3 and Pass@3 over multiple trials. Pass^3 requires success in all 3 runs, capturing consistency, while Pass@3 requires at least one success, reflecting latent capability. Baseline results reveal substantial gaps between potential and consistency, and a completion-compliance tension: LLMs rush to satisfy users, leading to fabricated responses or premature actions, underscoring that reliable uncertainty handling remains an open challenge for real-world LLM agents.

Testing work flow.

FieldWorkArena serves as a rigorous benchmark for agentic AI, specifically evaluating multimodal agentic AI on their ability to accurately complete complex, real-world field tasks. The benchmark's tasks are meticulously designed to simulate practical challenges in environments such as factories, warehouses and retails. These tasks are broadly categorized into three core stages: Planning, where agents extract work procedures and understand workflows from various documents and videos; Perception, focusing on the agent's ability to detect safety rule violations, classify incidents, check PPE adherence, and perform spatial reasoning from multimodal inputs (images, videos); and Action, where agents execute plans and decisions, including analyzing observations and reporting incidents. Additionally, Combination Tasks integrate these stages, requiring the agent to perform multi-step operations like detecting incidents from videos/documents and reporting them. Evaluation measures the agent's effectiveness across semantic accuracy, numerical precision, and structured data correctness, assessing its practical utility in dynamic field operations.