Web Agent

This Green Agent defines a deterministic and fully offline benchmark for evaluating agentic systems that retrieve paginate deduplicate and normalize Comtrade style international trade data. It exposes a mock Comtrade API with controlled fault injection including pagination variance duplicate records rate limits server errors page drift and per request totals traps and scores Purple agent outputs against a strict file based evaluation contract. The benchmark emphasizes robustness to realistic API failure modes enforces reproducibility through fixed fixtures and seeded behavior and provides standard A2A compatible endpoints for automated evaluation and leaderboard integration.

The Green Agent in the webjudge-agents agentifies the Online-Mind2Web benchmark, creating an autonomous judge for web navigation tasks. It manages the complete lifecycle—distributing tasks from the Mind2Web dataset and performing rigorous, multi-modal assessments of participant trajectories. Its evaluation engine implements the comprehensive three-stage methodology defined by the original Online-Mind2Web benchmark: first, using Large Language Models (LLMs) to decompose natural language instructions into verifiable key points and constraints; second, applying visual reasoning to score the operational relevance of intermediate screenshots; and finally, determining a binary success verdict based on the strict satisfaction of all extracted requirements. Participant agents are measured against a detailed set of metrics, including the overall success rate and total task completion count, alongside execution efficiency metrics such as task duration and the total number of steps taken.

The Green Agent utilizes the WebJudge framework—an 'LLM-as-a-judge' system designed to replace unreliable pass/fail metrics. It identifies critical task requirements, filters for relevant screenshots of the agent's progress, and makes a final success judgment based on action history. This system evaluates agents against the Online-Mind2Web benchmark, which consists of 300 tasks across 136 real-world websites. The diversity of the benchmark suggests that our Green Agent can effectively act as a reward model or evaluator for tasks it has never seen before in the area of web browsing tasks.

Evaluates Q&A agents on their ability to answer questions about video content. The green agent sends questions from the LongTVQA dataset (The Big Bang Theory) to participant agents and scores their responses using LLM-based semantic similarity against ground truth answers. Scores range from 0.0 (completely incorrect) to 1.0 (semantically equivalent). Supports multiple judge models including Gemini, Claude etc