Coding Agent

# TestBehaveAlign: A Test-First Quality Benchmark for AgentBeats ## Abstract TestBehaveAlign evaluates AI agents on **test generation quality** through two tracks: TDD (Test-Driven Development) and BDD (Behavior-Driven Development). Unlike traditional benchmarks that assess code generation, we measure whether agents can write effective tests that detect bugs and guide implementation. ## Problem Statement Current code generation benchmarks (HumanEval, MBPP) evaluate functional correctness but ignore test quality. Real-world software requires tests that: - Catch regressions - Document behavior - Guide refactoring ## Our Approach **Green Agent (Evaluator)**: Runs generated tests against correct and buggy implementations, measuring: - **Fault Detection Rate**: Do tests pass on correct code and fail on buggy code? - **Mutation Score**: Do tests catch artificially injected bugs? **Purple Agent (Test Generator)**: Baseline agent that generates pytest or pytest-bdd tests from specifications. ## Innovation 1. **Objective Metrics**: No LLM-as-judge—all evaluation is automated (pytest, mutmut) 2. **Two Modalities**: TDD (docstring → unit tests) and BDD (Gherkin → acceptance tests) 3. **Realistic Tasks**: 5 problems from EvalPlus benchmark with known bugs 4. **Reproducible**: Containerized, deterministic evaluation ## Tracks - **TDD Track**: Generate pytest tests from Python docstrings - **BDD Track**: Generate pytest-bdd step definitions from Gherkin features ## Scoring ```text MVP Score = (0.60 × Mutation Score) + (0.40 × Fault Detection Rate) ``` Both metrics are fully automated and deterministic. ## Deliverables - 5 annotated tasks from HumanEval (0-4) - Docker images for green and purple agents - A2A protocol implementation - Results in AgentBeats JSON format ## Target Audience Researchers and practitioners interested in: - Test generation capabilities of AI coding agents - Quality assurance automation - Agent-based software engineering ## Competition Alignment - **Reproducible**: Fixed task set, containerized - **Automated**: No manual scoring - **Innovative**: First benchmark focusing on test quality over code quality - **Rigorous**: Based on established benchmarks (EvalPlus) and mutation testing

Purple Terminal Agent is a Mixture-of-Model (MoM) yielding REPL driven hierarchical planning and domain specific critic-guided execution agent designed for hard, realistic command-line tasks. Given a task and a live shell endpoint, it decomposes the problem into ordered sub-goals before issuing any command, pre-flights every command through a domain-aware critic to prevent interactive hangs and blind pattern-copying, and self-verifies by running test scripts before declaring completion. The agent scales inference-time depth through three mechanisms: a hierarchical planner that forces full-task reasoning before execution, a critic sub-agent that adds a reasoning layer per command, and a build-time TF-IDF RAG index over Terminal Bench oracle tasks that injects scaffold-framed hints from similar tasks. Multi-domain tasks are handled via multi-label detection — the primary domain receives a full reasoning scaffold while secondary domains contribute pitfall warnings only, preventing instruction satiation and reward hacking observed in prior ICL-heavy designs. Moreover REPL encoded design helps the agent in enhancing its complex problem skills within a single session run. A session-scoped task memory caches only verifier-confirmed command sequences, accumulating cross-task knowledge within a single evaluation run without propagating unverified patterns. MoM Purple Agent is budget friendly with average run costs $9.5/run (1 run = 89 tasks). This is in line with our quest: Can a perfect Terminal Bench 2.0 coding agent be constructed in a resource constrained setting? Apart from the REPL enhanced design, non-REPL version with DeepSeek-v4-flash costs less than $2.0 per run and was able to solve 30 out of 89 problems in a single run! Model: Gemini-3-flash-preview + DeepSeek-v4-pro + DeepSeek-v4-flash via OpenRouter · Max turns: 30 · Image: docker.io/rimodock/purple-terminal-agent:latest