Coding Agent

This green agent implements a software debugging benchmark that evaluates purple agents on their ability to identify, analyze, and fix real-world software bugs. The benchmark uses three established bug repositories: Defects4J for Java, BugsJS for JavaScript, and BugsInPy for Python. These repositories contain authentic bugs from production codebases, providing realistic debugging challenges across multiple programming languages. The benchmark evaluates four core capabilities. First, agents must localize bugs by identifying which source files and code regions contain defects. They do this by analyzing failing test cases and their outputs. Second, agents need to perform root cause analysis to understand why tests fail. This involves examining error messages, stack traces, and the relationship between buggy code and test expectations. Third, agents must generate patches that fix the identified bugs without breaking existing functionality. Fourth, agents should verify their fixes by ensuring that previously failing tests now pass and that no new test failures are introduced. The evaluation process follows a consistent workflow. For each bug instance, the green agent checks out both buggy and fixed versions of the code, compiles the project, and runs the test suite. It provides the purple agent with information about failing tests and evaluates proposed fixes by applying patches and rerunning tests. Scoring is based on test pass rates, code coverage, and patch quality. This multi-language approach tests whether agents can demonstrate debugging skills that work across different programming languages while handling the specific challenges of each ecosystem, including different build systems, testing frameworks, and language conventions.

Our green evaluator agent benchmarks database-focused agents on Spider2-Snow, a suite of natural-language-to-SQL tasks grounded in Snowflake-backed datasets. For each test instance, it provides the target agent with the instruction, db_id, and any optional external knowledge, and expects a structured response containing a single SQL query (via an A2A DataPart like {"sql": "..."}; plain-text and fenced ```sql fallbacks are also supported). The evaluator then executes the predicted SQL on Snowflake and compares the resulting output to gold execution results to score correctness.

Our green agent implements an A2A-compatible evaluator for the Data Analysis Benchmark (DABench), a benchmark designed to assess LLM-based agents on realistic data analysis tasks over CSV datasets. DABench defines end-to-end analytical questions that require agents to interpret data, perform transformations, and produce verifiable outputs, enabling systematic evaluation of data analysis capabilities (see DABench paper: https://arxiv.org/html/2401.05507v1). Within this setup, the green agent (1) loads and structures tasks from the DABench benchmark, (2) dispatches clear analytical instructions to a participating agent via the A2A protocol, and (3) evaluates the agent’s responses using an LLM-as-judge approach to assess correctness and completeness. The green agent focuses exclusively on orchestration and evaluation, while reasoning and code execution are fully handled by the participating agent.

Autonomous coding agents are increasingly expected to solve complex, real-world API tasks involving multiple services, dependencies, and alternative solution paths. However, most existing benchmarks, including SOCBench-D, implicitly assume simplified one-to-one task–solution mappings and lack support for evaluating agentic behavior in realistic many-to-many (n:m) settings. As a result, current evaluations fail to capture whether an agent truly understands which APIs are required, how they should be combined, and which endpoints should be avoided. We present SOCBench Runner, a Green Agent that transforms SOCBench-D, a benchmark for evaluating automated REST API integration coding, into a fully agentic, reproducible benchmark within the AgentBeats platform. The Green Agent orchestrates evaluations for multiple Purple Agents that autonomously generate Python code to solve natural-language API tasks. Instead of relying solely on execution success, our approach performs static code analysis to extract all referenced API endpoints and evaluates performance using precision, recall, and F1 scores over task-specific ground-truth API sets. The benchmark supports a wide range of scenarios, including graded difficulty levels (easy, medium, hard), retrieval-augmented generation (RAG) settings, and real-world REST API tasks adapted from RestBench. This design enables fine-grained measurement of endpoint selection accuracy, coverage, overuse, and task completion across diverse domains. By agentifying SOCBench-D and explicitly targeting the n:m task–API evaluation gap, our framework establishes a standardized and extensible benchmark for autonomous coding agents. It provides actionable insights into agents’ ability to reason about API ecosystems, retrieve relevant specifications, and generate correct, efficient code, advancing the evaluation of LLM-driven software development in realistic, production-oriented settings.

The rapid growth of scientific literature has made it increasingly difficult for researchers to efficiently discover, evaluate, and synthesize relevant work. Recent advances in multi-agent large language models (LLMs) have demonstrated strong potential for understanding user intent and are being trained to utilize various tools. In this paper, we introduce Paper Circle, a multi-agent research discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature. The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-criteria scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes (e.g., concepts, methods, experiments, and figures) and edges, enabling graph-aware question answering and coverage verification. Both pipelines are implemented within a coder LLM–based multi-agent orchestration framework and produce fully reproducible, synchronized outputs (JSON, CSV, BibTeX, Markdown, and HTML) at each agent step. This paper describes the system architecture, agent roles, retrieval and scoring methods, knowledge graph schema, and evaluation interfaces that together form the Paper Circle research workflow. We benchmark Paper Circle on both paper retrieval and paper review generation, reporting hit rate, MRR, and Recall@K. Results show consistent improvements with stronger agent models. Github: https://github.com/MAXNORM8650/papercircle

Evaluate an agent’s ability to solve USACO programming problems, including reasoning through complex algorithmic challenges and designing novel solutions under strict time and memory constraints.