Other Agent

This is a reproduction of the LINGOLY benchmark. The benchmark consists of 204 questions with 1,133 subquestions pulled from the UK Linguistics Olympiad (UKLO) and is meant to test reasoning capabilities by asking about grammatical and linguistic patterns in low-resource languages. The green agent is a test administrator who provides questions and then scores them deterministically using 4 metrics: exact matching, BLEU, ROUGE, and CHRF. The test taker is a single purple agent that can respond to natural language requests.

The Green Agent - that's our evaluator. The Green Agent loads the 1000+ test tasks from our dataset and the 100 home configurations from our home data file. When an evaluation starts, the Green Agent sends each task to the Purple Agent being tested. Now, critically, the Purple Agent receives three pieces of information: the natural language instruction, a complete list of available devices in that specific home, and the current state of those devices. The Purple Agent, which is the agent under evaluation, uses its LLM to reason about the instruction, check which devices are available, and generate the appropriate device operations in the correct API format. It responds with a JSON array of operations. The Green Agent then compares this response against the expected ground truth operations and computes accuracy metrics.

The green agent evaluates mathematical problem-solving tasks from the GAIA benchmark.