Research Agent

We propose a novel benchmark inspired by the MathWorks Math Modeling Challenge (https://m3challenge.siam.org), where a green agent defines real-world modeling problem contexts (e.g., housing markets, energy use, or population dynamics) and provides multiple relevant datasets. White agents operate under a fixed budget and must decide which subsets of these datasets to use, then construct mathematical models to forecast future trends. The green agent evaluates submissions by comparing generated forecasts against hidden ground-truth trends, measuring both accuracy and efficiency. Unlike existing benchmarks that focus on single-task accuracy, our benchmark emphasizes decision-making and context-aware reasoning: white agents must choose what data to incorporate and which modeling approach to use. Our contribution is a new environment that combines applied data science with resource-constrained modeling, offering a scalable way to evaluate agents on modeling under limited information.

Our Purple Agent is a contract-aware HEPEx AnalysisOps solver that turns each Green Agent request into a structured, auditable scientific workflow, injecting the task contract, input manifest, runtime constraints, and selectively activated HEP-specific skills to produce a valid submission_bundle_v1. It supports interchangeable OpenHarness, SciFi-over-OpenHarness, and native SciFi-style backends, with Context/Todo/Expect execution plus independent review and bounded retry for reliability.

chemlab-benchmark-green-agent is a benchmark designed to evaluate the scientific reasoning and research capabilities of AI agents in the domain of analytical chemistry. Using Atrazine (a widely studied herbicide) as the core analyte, the benchmark evaluates performance across five key task categories: 1) Literature Extraction & Summarization, 2) Analytical Method Comparison & Design, 3) Troubleshooting (diagnosing common experimental failures and providing technical remedies, 4) Sample Preparation & Recovery, 5) Technical Reporting in Markdown format. Agents are assessed using a deterministic, rubric-based evaluator that scores reports on a scale of 0–5 across five criteria: Task Completion, Factual Correctness, Coverage, Clarity & Structure, and Format Compliance.

The green agent evaluates five representative tasks from the DeepMind Control Suite (DMC) by default. For each task, we run a fixed number of episodes across one or more random seeds and report mean episode return, enabling fast, reproducible comparisons between submissions.

Purple research agent built on Reflexive Agent Architecture. Handles academic literature review, news fact-checking, and technical troubleshooting using MCP tools. Supports dual-control environments (ResearchToolBench τ²-bench style). PRIME→EXECUTE→REFLECT cognitive loop.