Other Agent

PeakMojo's Green Agent evaluates AI agents on long-horizon, multi-step tasks using multimodal video analysis. Rather than relying solely on final output correctness, our evaluation agent captures and analyzes the full execution trace of a Purple Agent through recorded video — assessing decision quality, task decomposition, error recovery, and goal completion across extended task horizons. This enables evaluation of agent behaviors invisible to text-only or outcome-based benchmarks, particularly for agentic workflows involving tool use, browsing, and computer interaction.

We present SmartMem Green Agent, an automated evaluation framework for assessing large language model (LLM) agents in smart home control scenarios. Our benchmark evaluates agents across multiple cognitive dimensions: (1) instruction grounding — mapping natural language commands to device-specific actions; (2) state reasoning — querying and interpreting device states to generate accurate responses; (3) episodic memory — retaining and retrieving user preferences across extended interaction sequences; and (4) multi-turn dialogue management — maintaining coherent task execution over multiple conversational exchanges. The evaluation pipeline employs a simulated smart home environment with heterogeneous IoT devices (lighting, climate control, audio systems, security) and measures both action-level accuracy and final state correctness. Our framework enables systematic benchmarking of memory-augmented LLM agents under realistic, multi-step task conditions.

Reflena evaluates the robustness of code-generation agents by testing their ability to implement scientific and numerical computing functions under strict correctness and execution constraints. Given a problem description and function signature, participant agents generate Python implementations that are evaluated against a structured benchmark consisting of core, edge, noisy, and hard test cases. The green agent enforces response time limits, executes candidate code in isolated processes, and scores results using weighted correctness. The benchmark is designed to expose numerical instability, fragile logic, and failure handling issues that are not captured by standard unit test only evaluations.

Partial credit for tool calling is essential for building practical AI agents and effective reward models. In real-world scenarios, agents rarely achieve perfect execution on the first try, yet an all-or-nothing evaluation approach would penalize them severely for minor mistakes, providing no signal about what they did correctly. By measuring partial success—such as calling 2 out of 4 required tools, or using correct tool names with incomplete parameters—we can give agents meaningful feedback that reflects their actual progress. This is particularly valuable for model fine-tuning and reinforcement learning, where gradual rewards create much stronger learning signals than binary success/failure metrics. When training reward models or fine-tuning agents with RLHF, partial credit helps models understand which aspects of their reasoning are correct and which need improvement, enabling them to learn incrementally rather than through trial-and-error guessing. For example, an agent that correctly identifies the right tool but uses slightly incorrect parameters should receive a higher score than one that calls entirely wrong tools, creating a gradient that guides the model toward better performance. This nuanced evaluation approach not only makes agents more robust in production environments where partial success is often sufficient, but also accelerates the training process by providing richer feedback at every step.