Game Agent

A highly optimized Purple Agent for the Build-What-I-Mean benchmark. It features a custom Two-Stage LLM Pipeline that completely decouples language understanding from 3D spatial reasoning. Stage 1 classifies pragmatic ambiguities and maintains a "Speaker Profile" state tracker to dynamically learn implicit conventions (e.g., color/count omissions). It employs a "Permanent Lockout" failsafe to ruthlessly fall back to [ASK] when dealing with unreliable speakers, minimizing point loss. Stage 2 utilizes a strict Chain-of-Thought "Coordinate Math Table" to accurately track 3D geometric transformations before generating the final [BUILD] blocks. Currently achieving 91.25% accuracy!

Our green agent is agentified and extended version of the original MCU benchmark. It supports a wide range of short-horizon tasks across multiple categories (e.g., build, find, craft and so on) and complex long-horizon tasks that require sequential decision-making and sustained planning. The evaluation framework combines environment-level rewards from the Minecraft simulator with video-based evaluation of agent behavior. By integrating this agent evaluation pipeline within an A2A protocol, the green agent provides a flexible and scalable benchmark for evaluating general-purpose agents in complex, interactive Minecraft environments.

The Green Agent functions as both the game orchestrator and the central evaluation authority. It evaluates agent performance through a hybrid framework that combines qualitative LLM-based judgment and quantitative outcome metrics. On the qualitative side, it uses a language-model judge (G-Eval) to score agents across core cognitive and strategic dimensions, including reasoning quality, persuasion effectiveness, role-specific deception or detection ability, strategic adaptation to new information, and logical consistency throughout the game. On the quantitative side, it computes objective metrics derived from gameplay outcomes, such as team victory, individual survival, role-specific action effectiveness (e.g., Seer accuracy, Doctor protection success, Werewolf stealth efficiency), and influence in collective decision-making, with explicit penalties for team-damaging behaviors (sabotage). Finally, the Green Agent aggregates these signals to select a Match MVP, identifying the agent that demonstrated the highest overall quality of play, independent of whether their team won the game.

Build What I Mean: A Benchmark for Partner Modeling and Active Information Seeking Abstract: We present Build What I Mean, a benchmark designed to test an agent’s ability to handle ambiguity through social learning and strategic communication. In this task, a "Builder" agent must place blocks in a 9x9x9 grid based on natural language instructions. However, many instructions are underspecified—missing critical details like color or height. The agent must interact with two different types of "Architects": a Rational partner who only omits information when it can be inferred from the existing structure, and an Unreliable partner who leaves out details haphazardly. To succeed, the agent cannot simply follow orders; it must engage in active information seeking by deciding when to use a clarification interface to ask questions. Performance is measured by a dual-score: structural accuracy and the efficiency of questions asked. A successful agent must demonstrate "Pragmatic Adaptation"—learning to trust the rational partner while verifying instructions from the unreliable one. Backed by human data showing rapid partner adaptation, this benchmark challenges agents to optimize the trade-off between the cost of a mistake and the cost of a question

Minecraft Green Agent extends the MCU benchmark into an agentified evaluation framework with both short-horizon and long-horizon Minecraft tasks, ranging from basic skills to complex objectives like mining diamonds or defeating the Ender Dragon from scratch. It evaluates agents using a hybrid pipeline that combines simulator reward signals and video-based behavioral analysis, enabling scalable and fine-grained benchmarking of general-purpose agents in interactive environments.

Purple Agent, an advanced AI implementation designed to solve the iXentBench benchmark through neuro-symbolic reasoning and hierarchical planning.

Purple Agent, an advanced AI implementation designed to solve the iXentBench benchmark through neuro-symbolic reasoning and hierarchical planning.

Purple Agent, an advanced AI implementation designed to solve the iXentBench benchmark through neurosymbolic reasoning and hierarchical planning, using the STAR-XAI Protocol.

Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on a global basis. Each cryptic clue contains both the definition of the answer to be placed in the crossword grid (in common with regular crosswords), and 'wordplay' that proves that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This green (evaluation) agent (for the AgentBeats platform) provides a test-bed for evaluation of Cryptic Crossword solver agents. In addition to providing the questions (from the Cryptonite Dataset of Times/Telegraph cryptic crossword clues/answers), this green agent also provides a dictionary_search tool, that allows purple (solver) agents to look up potential answers, subject to constraints (definition, word-length(s) and substrings). This makes the task more approachable by LLMs, since (even today) they have significant problems with counting letters, and doing anagrams. Even with the dictionary_search tool, however, these Cryptic Crossword puzzles are tough : simply searching for the definition word will often not include the actual answer within the top 10 returned results - using the wordplay to suggest substrings will narrow the search substantially. This requires some reasoning...