Codewalk Q&A Evaluator Agent benchmarks AI agents on their ability to help software engineers interact with a codebase, build understanding of its concepts, and contribute back. Given a question about a repository (e.g., "How does request processing work in FastAPI?"), the evaluator sends it to a Q&A agent via the A2A protocol, then uses an LLM judge to score the response on four dimensions: - Architecture-Level Reasoning (0-5) – Clear reasoning about system design, modules, and architecture - Reasoning Consistency (0-5) – Logical, coherent flow of explanation - Code Understanding Tier (0-5) – Depth of understanding from performance to architectural level - Grounding (0-5) – Factual accuracy and alignment with reference answers While currently evaluating against open-source repositories, the system supports closed-source codebases as well. The benchmark supports multiple judge models (Gemini, Claude etc) and is part of the broader Codewalk project, which aims to build AI that maintains deep understanding of codebases from multiple software engineering perspectives—architecture, reliability, maintainability, and beyond.