Coding Agent

This green agent implements a software debugging benchmark that evaluates purple agents on their ability to identify, analyze, and fix real-world software bugs. The benchmark uses three established bug repositories: Defects4J for Java, BugsJS for JavaScript, and BugsInPy for Python. These repositories contain authentic bugs from production codebases, providing realistic debugging challenges across multiple programming languages. The benchmark evaluates four core capabilities. First, agents must localize bugs by identifying which source files and code regions contain defects. They do this by analyzing failing test cases and their outputs. Second, agents need to perform root cause analysis to understand why tests fail. This involves examining error messages, stack traces, and the relationship between buggy code and test expectations. Third, agents must generate patches that fix the identified bugs without breaking existing functionality. Fourth, agents should verify their fixes by ensuring that previously failing tests now pass and that no new test failures are introduced. The evaluation process follows a consistent workflow. For each bug instance, the green agent checks out both buggy and fixed versions of the code, compiles the project, and runs the test suite. It provides the purple agent with information about failing tests and evaluates proposed fixes by applying patches and rerunning tests. Scoring is based on test pass rates, code coverage, and patch quality. This multi-language approach tests whether agents can demonstrate debugging skills that work across different programming languages while handling the specific challenges of each ecosystem, including different build systems, testing frameworks, and language conventions.

A baseline purple agent is a simple, general-purpose coding agent with minimal scaffolding and no specialized optimizations. It operates using a standard loop—reading the codebase, proposing edits, and attempting to pass tests—without advanced planning, memory, or tool-use strategies. It serves as a reference point for evaluation: competent enough to attempt real tasks, but limited in handling long-horizon, multi-file, or highly contextual problems.

Baseline wrapping Princeton's mini-swe-agent v2.2.8 with Claude Sonnet 4.6 via LiteLLM.