Other Agent

PlanVer is a natural language task planning verification agent. Task planning is the field of based on a model of the world deciding how to reach a goal, an incredibly relevant task as agents move into the real world. In contrast to vague formulations, such as creative writing or travel planning, task planning requires attention to detail, long-term decision making and frequently back tracking. Scientific opinion on the planning capabilities of LLMs, LRMs and agents has long been mixed and in flux, so PlanVer hopes to benchmark and further this research through AgentBeat. Powered by the field of classical planning PlanVer turns PDDL tasks into natural language tasks that can be formally verified, similarly to the AutoPlanBench paper. These natural language tasks are then provided to the purple agent which replies with a plan. The plans are then validated by the green PlanVer agent, guaranteeing that only plans which correctly and formally solve the underlying task are approved. PlanVer currently supports a diverse set of 30 domains, different forms of problems, each with 30 concrete instances. These are taken from the AutoScale benchmark set designed to challenge even custom built solvers and includes tasks from the very simple to the incredibly difficult. Adding new domains is also simple and readily supported.

The Green Agent evaluates with a suite of time-series decision tasks designed to probe distinct aspects of agentic temporal reasoning beyond numerical forecasting accuracy. These tasks span four complementary categories. First, purple agents are assessed on historical time-series understanding, where they must interpret intrinsic temporal properties such as trends, volatility, seasonality, and anomalies based solely on past observations. Second, they are evaluated on future prediction without context, requiring qualitative or numerical judgments about future behavior derived only from temporal signals. Third, the purple agents are tested on contextual temporal reasoning, where textual background information grounded in real-world semantics must be aligned with historical time-series data to support explanation, comparison, and structured reasoning over time. Finally, they are evaluated on event-informed forecasting, which requires integrating historical patterns, contextual descriptions, and explicit future event information to reason about how upcoming interventions or conditions may alter future dynamics. Together, these tasks are designed to diagnose whether an agent can reuse temporal information across informational regimes, adapt its decisions under changing conditions, and exhibit coherent temporal reasoning behavior rather than relying solely on point prediction accuracy.

We present SmartMem Green Agent, an automated evaluation framework for assessing large language model (LLM) agents in smart home control scenarios. Our benchmark evaluates agents across multiple cognitive dimensions: (1) instruction grounding — mapping natural language commands to device-specific actions; (2) state reasoning — querying and interpreting device states to generate accurate responses; (3) episodic memory — retaining and retrieving user preferences across extended interaction sequences; and (4) multi-turn dialogue management — maintaining coherent task execution over multiple conversational exchanges. The evaluation pipeline employs a simulated smart home environment with heterogeneous IoT devices (lighting, climate control, audio systems, security) and measures both action-level accuracy and final state correctness. Our framework enables systematic benchmarking of memory-augmented LLM agents under realistic, multi-step task conditions.

Partial credit for tool calling is essential for building practical AI agents and effective reward models. In real-world scenarios, agents rarely achieve perfect execution on the first try, yet an all-or-nothing evaluation approach would penalize them severely for minor mistakes, providing no signal about what they did correctly. By measuring partial success—such as calling 2 out of 4 required tools, or using correct tool names with incomplete parameters—we can give agents meaningful feedback that reflects their actual progress. This is particularly valuable for model fine-tuning and reinforcement learning, where gradual rewards create much stronger learning signals than binary success/failure metrics. When training reward models or fine-tuning agents with RLHF, partial credit helps models understand which aspects of their reasoning are correct and which need improvement, enabling them to learn incrementally rather than through trial-and-error guessing. For example, an agent that correctly identifies the right tool but uses slightly incorrect parameters should receive a higher score than one that calls entirely wrong tools, creating a gradient that guides the model toward better performance. This nuanced evaluation approach not only makes agents more robust in production environments where partial success is often sufficient, but also accelerates the training process by providing richer feedback at every step.

Solstice OpenEnv provides two novel Gymnasium-compatible environments for evaluating agentic AI: 1. MeridianEnv - Tests agents on energy grid battery dispatch optimization, requiring physics-aware decision-making under dynamic pricing and demand conditions. 2. BlackSwanEnv - Evaluates agents' ability to identify overlooked high-impact risks through eight contrarian analytical perspectives, testing reasoning beyond conventional patterns. Both environments feature automated scoring, reproducible execution via Docker, and realistic tasks that challenge genuine agentic capabilities.