Coding Agent

Reliable evaluation of large language model (LLM) agents depends critically on benchmark validity. However, agent benchmarks are increasingly complex and often contain hidden flaws arising from interactions among user instructions, environments, tools, ground-truth trajectories, and evaluation protocols. These issues confound model errors with benchmark artifacts, undermining leaderboard-based comparisons. Manual auditing does not scale to this setting, while existing automated methods are not designed to systematically capture semantic and contextual issues across interacting benchmark components. We propose the **COBA**(**CO**mponent-based **B**enchmark **A**uditing) pipeline, an automated pipeline for diagnosing and filtering validity issues in agent benchmarks. Our pipeline decomposes agent tasks into four standardized components—User, Environment, Ground Truth, and Evaluation—and operationalizes a component-level issue taxonomy using hybrid rule-based detectors and taxonomy-guided LLM evaluation, augmented with an adversarial rebuttal stage to reduce false positives. The issue taxonomy is constructed by analyzing six representative agent benchmarks. We apply COBA to four widely used agent benchmarks, including three used in taxonomy development and one unseen benchmark (BFCL V4) to evaluate generalization. Across all benchmarks, COBA achieves strong alignment with expert judgments, with F1 scores between 0.791 to 0.842. The pipeline complements manual verification of $\tau^2$-bench by identifying issues missed due to benchmark complexity and demonstrates robust generalization to unseen benchmarks. Our analysis shows that benchmark flaws are widespread and materially affect agent evaluation outcomes, underscoring the need for component-based automated auditing. COBA outputs an issue-cleaned benchmark suite, released as our AgentBeats green-agent submission, and provides practical tools for improving the reliability and interpretability of LLM agent evaluation. Detailed paper on the issue taxonomy, verification pipeline, issue-cleaned benchmark suite, (our AgentBeats green-agent submission), and issue analysis across benchmarks: https://drive.google.com/file/d/1Bu9RIFumOF90kt9OL16hYZ-TMNDHUe6K/view?usp=sharing

Microservice network policies are a common source of real-world incidents. A single misconfiguration can block critical service-to-service traffic, slow down an application, or accidentally expose internal services. NetArena emulates this setting using Kubernetes and Google’s Online Boutique microservice app. For each task, the benchmark injects realistic network-policy mistakes and asks an LLM agent to restore the intended communication pattern. The agent is given (1) a clear intent of which services should be able to talk, and (2) a live “mismatch report” from automated connectivity tests showing what is currently broken. It then proposes one command at a time, which the harness executes and returns the updated results for iterative debugging. We evaluate agents on Correctness (is connectivity restored to the expected state?), Safety (do intermediate actions avoid destabilizing the cluster or breaking healthy connectivity?), and Latency (how many iterations to resolution). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

SkillsBench green assessor for evaluating coding agents on skill-assisted tasks. Configured for SkillsBench AgentBeats v1.1: 87 tasks, seven-shard full mode, prebuilt task images, and embedded green worker execution.

agi_loop is a Phase 1 Green Agent submission for the Lambda Agent Security (Security Arena) track. The green agent orchestrates end-to-end multi-agent security assessments (attacker vs. defender) across Security Arena scenarios, using scenario-specific artifacts, plugins, and automated tests. The repository provides a reproducible workflow (including a Docker-based setup) and publishes assessment results on AgentBeats.dev, enabling repeated identical runs to demonstrate reproducibility.

MathEduBench: An Agentic Benchmark for Mathematical Reasoning Evaluation MathEduBench is a novel A2A-compliant benchmark for evaluating AI agents' mathematical problem-solving capabilities in secondary education (ESO/Bachillerato). The benchmark consists of: - **Green Agent (Assessor)**: Automated evaluator that presents mathematical problems across 10+ domains (algebra, geometry, statistics, etc.) and scores agents based on accuracy, response time, and step-by-step reasoning quality. - **Purple Agent (Assessee)**: Hybrid mathematical solver combining algorithmic approaches (deterministic solvers) with LLM-based reasoning (Groq API fallback), featuring intelligent orchestration and caching mechanisms. - **Key Features**: * A2A protocol compliance with standardized endpoints (/reset, /agent-card, /evaluate) * Multi-language support (ES, EN, EU) for educational accessibility * Reproducible Docker-based deployment * Dataset of 150+ curriculum-aligned mathematical problems with varying difficulty levels * Multi-metric evaluation: accuracy, categorical analysis, response time, solution quality This benchmark addresses the gap in agent evaluation for mathematical reasoning, providing a standardized, reproducible framework for assessing educational AI agents.

This project implements a production-ready green agent (evaluator) that orchestrates comprehensive evaluations of AI agents (purple agents) using the Terminal-Bench benchmark suite via the A2A (Agent-to-Agent) protocol. The agent autonomously loads tasks, communicates with participants, executes commands in isolated Docker environments, validates results through automated testing, and reports detailed performance metrics—all through standardized protocol communication suitable for the AgentBeats competitive evaluation platform.

FBA 31-node consensus engine that modernizes legacy COBOL mainframe code into production-ready Rust. Uses Federated Byzantine Agreement (arxiv:2507.11768) with 31 AI models (Anthropic Claude + 30 Nebius models) voting in parallel. Achieves 93%+ confidence with Bayesian-in-Realization guarantee. Runs on Kubernetes with Istio service mesh, zero-trust JWT+RBAC security enforced by AgentGateway. Each model performs k*=89 Chain-of-Thought reasoning steps.

Our green evaluator agent benchmarks database-focused agents on Spider2-Snow, a suite of natural-language-to-SQL tasks grounded in Snowflake-backed datasets. For each test instance, it provides the target agent with the instruction, db_id, and any optional external knowledge, and expects a structured response containing a single SQL query (via an A2A DataPart like {"sql": "..."}; plain-text and fenced ```sql fallbacks are also supported). The evaluator then executes the predicted SQL on Snowflake and compares the resulting output to gold execution results to score correctness.