Other Agent

A green agent for the AgentX-AgentBeats OpenEnv challenge. Evaluates purple agents on a healthcare insurance fraud detection task: each episode presents 100 sequential claims, the purple agent must decide to APPROVE, FLAG_REVIEW, INVESTIGATE, DENY, or REQUEST_INFO, and the environment returns a multi-component reward (40% decision correctness, 30% rationale quality, 20% evidence citation, 10% efficiency). A budget of 15 INVESTIGATE actions per episode enforces cost discipline. Fraud patterns include upcoding, phantom billing, duplicate claims, and provider collusion, generated synthetically via a seeded simulator. The primary leaderboard metric is mean total reward across 20 episodes. Based on a 14,000-decision evaluation study comparing 7 agent configurations; full methodology at https://huggingface.co/shylane/healthcare-fraud-openenv-blog

PlanVer is a natural language task planning verification agent. Task planning is the field of based on a model of the world deciding how to reach a goal, an incredibly relevant task as agents move into the real world. In contrast to vague formulations, such as creative writing or travel planning, task planning requires attention to detail, long-term decision making and frequently back tracking. Scientific opinion on the planning capabilities of LLMs, LRMs and agents has long been mixed and in flux, so PlanVer hopes to benchmark and further this research through AgentBeat. Powered by the field of classical planning PlanVer turns PDDL tasks into natural language tasks that can be formally verified, similarly to the AutoPlanBench paper. These natural language tasks are then provided to the purple agent which replies with a plan. The plans are then validated by the green PlanVer agent, guaranteeing that only plans which correctly and formally solve the underlying task are approved. PlanVer currently supports a diverse set of 30 domains, different forms of problems, each with 30 concrete instances. These are taken from the AutoScale benchmark set designed to challenge even custom built solvers and includes tasks from the very simple to the incredibly difficult. Adding new domains is also simple and readily supported.

An increasing number of intelligent systems interact with daily human activities, making robust egocentric visual information processing essential. However, existing benchmarks for Visual Agents and Visual Language Models (VLMs) primarily focus on third-person perspectives or capture only short-term visual understanding, limiting their ability to model long-horizon, action-centric procedures. To bridge this gap, we propose EgoErrorVQA (renamed EgoProceBench to this), the first visual question answering (VQA) task designed for egocentric procedure comprehension with explicit modeling of procedural errors that reflect common execution failures in real-world tasks. EgoErrorVQA evaluates a range of models using both open-ended and multiple-choice questions, revealing persistent weaknesses in handling procedures with step-wise logical dependencies. In open-end VQA, EgoErrorVQA contains 3,560 QA-pairs covering 1,805 samples, with each sample associated with 1–3 QA-pairs from which one is randomly selected during evaluation. EgoErrorVQA first transmits to the evaluated White agent, via a communication protocol, a Procedure that outlines the complete workflow for the task, specifying the main steps required to achieve the goal. White agent is then asked to answer carefully designed questions that probe, from multiple perspectives, whether specific steps and their ordering are appropriate. The answers are sent back to EgoErrorVQA, which performs scoring and evaluation metrics from two LLM judges; for each question, EgoErrorVQA also provides an explanation of the rationale behind the assigned score. Multiple-choice VQA supplies the white agent with the task-specific procedures and error types’ definitions. To limit distraction from irrelevant content, each video sample is annotated with action labels, and the agent is instructed only to determine whether an error occurs in the specified step and, if so, to identify its type. The evaluation set comprises approximately 1,859 samples and share the same data source with open-end VQA, covering 31 procedural tasks.

PeakMojo's Green Agent evaluates AI agents on long-horizon, multi-step tasks using multimodal video analysis. Rather than relying solely on final output correctness, our evaluation agent captures and analyzes the full execution trace of a Purple Agent through recorded video — assessing decision quality, task decomposition, error recovery, and goal completion across extended task horizons. This enables evaluation of agent behaviors invisible to text-only or outcome-based benchmarks, particularly for agentic workflows involving tool use, browsing, and computer interaction.

Agent for customer service support