Research Agent

chemlab-benchmark-green-agent is a benchmark designed to evaluate the scientific reasoning and research capabilities of AI agents in the domain of analytical chemistry. Using Atrazine (a widely studied herbicide) as the core analyte, the benchmark evaluates performance across five key task categories: 1) Literature Extraction & Summarization, 2) Analytical Method Comparison & Design, 3) Troubleshooting (diagnosing common experimental failures and providing technical remedies, 4) Sample Preparation & Recovery, 5) Technical Reporting in Markdown format. Agents are assessed using a deterministic, rubric-based evaluator that scores reports on a scale of 0–5 across five criteria: Task Completion, Factual Correctness, Coverage, Clarity & Structure, and Format Compliance.

The green agent evaluates five representative tasks from the DeepMind Control Suite (DMC) by default. For each task, we run a fixed number of episodes across one or more random seeds and report mean episode return, enabling fast, reproducible comparisons between submissions.

The green agent evaluates research and web agents on long-horizon, multi-step reasoning tasks constructed through counterfactual expansion to expose jagged intelligence and weakness as task complexity increases. Tasks span information seeking, financial analysis, and scientific investigation, and require agents to sustain coherent reasoning over extended web-based and code-based trajectories. For each task, the underlying reasoning chain is systematically expanded to increase difficulty in a controlled manner. This design enables precise diagnosis of when and how a research or web agent fails within a long-horizon task, rather than only measuring final-task success.