Coding Agent

AgentBeats-ready green agent for the ArchXBench RTL synthesis benchmark. The service exposes the A2A-compatible agent card plus task discovery and health endpoints, and evaluates Verilog submissions with Icarus Verilog (and optionally Yosys for PPA metrics).

Our green evaluator agent benchmarks database-focused agents on Spider2-Snow, a suite of natural-language-to-SQL tasks grounded in Snowflake-backed datasets. For each test instance, it provides the target agent with the instruction, db_id, and any optional external knowledge, and expects a structured response containing a single SQL query (via an A2A DataPart like {"sql": "..."}; plain-text and fenced ```sql fallbacks are also supported). The evaluator then executes the predicted SQL on Snowflake and compares the resulting output to gold execution results to score correctness.

SWE-Bench Verified Green Agent - Task Description The green agent is an automated evaluator designed to assess the software engineering capabilities of participant agents. It uses the SWE-Bench Verified dataset, which contains real-world GitHub issues and their corresponding fixes from popular open-source Python repositories. What It Evaluates The green agent measures how well a participant agent can understand a problem description, analyze a codebase, and produce a working patch that fixes the reported issue—all without introducing regressions to existing functionality. Evaluation Workflow The evaluation process consists of seven phases: Phase 1: Repository Cloning and Initial Checkout — The green agent clones the target repository from GitHub and checks out the environment_setup_commit. This is a specific commit where the environment is known to be stable for dependency installation. Phase 2: Environment Setup — Using an LLM-powered agentic loop, the green agent reads the repository's configuration files (like pyproject.toml or setup.py) to identify the required Python version. It then installs that Python version, creates a virtual environment, and sets up pip. Phase 3: Dependency Installation — The agent installs all necessary dependencies, including build tools like setuptools and wheel, followed by the package itself with its test dependencies (typically using pip install -e ".[test]"). Phase 4: Switching to the Base Commit — After the environment is ready, the agent switches to the base_commit. This is the exact state of the codebase where the bug exists—before any fix was applied. Phase 5: Problem Dispatch to Participant — The green agent sends the problem statement and any hints to the participant agent via the A2A protocol. The participant agent is expected to analyze the problem, explore the codebase using available tools, and return a patch that fixes the issue. Phase 6: Patch Extraction and Application — Once the participant responds, the green agent extracts the patch from the response. It tries multiple application strategies including git apply, git apply --ignore-whitespace, git apply --3way, and patch -p1 with fuzz factors to handle common formatting issues that LLMs may produce. Phase 7: Test Execution and Scoring — The green agent runs two sets of tests. First, it runs the FAIL_TO_PASS tests—these are tests that were failing before the fix and should now pass if the participant's patch correctly addresses the issue. Second, it runs the PASS_TO_PASS tests—these are tests that were already passing before the fix and should continue to pass, ensuring the patch doesn't introduce regressions. Scoring and Final Status Based on the test results, each instance receives one of the following status classifications: Resolved means all FAIL_TO_PASS tests now pass and all PASS_TO_PASS tests continue to pass. This is the ideal outcome indicating a complete and correct fix. Breaking Resolved means the participant successfully fixed all the failing tests, but some previously passing tests now fail. The fix works but introduces regressions elsewhere in the codebase. Partially Resolved means some (but not all) of the failing tests now pass, while all previously passing tests continue to pass. The fix is incomplete but doesn't break anything.

DebateJudge-GreenAgent evaluates reasoning and argumentation tasks automatically.