Agentic Judges Boost Code LLM Reasoning 540% on SWE-bench
Fine-tuning Qwen3 on 3,360 curated instances yields 27.2% resolved rate
LLMs struggle with software architecture understanding, which is costly to label manually and impossible to verify through tests alone. To solve this, researchers introduced an agentic judging pipeline that uses a strong LLM as a scalable proxy for expert architectural evaluation. It comprises two specialized judges: the Architecture Complexity Judge (ACJ), which estimates the architectural understanding a task demands based on the codebase, and the Architecture Quality Judge (AQJ), which evaluates whether a patch conforms to repository-specific conventions using source-grounded rubrics.
To train the models, they curated 3,360 instances and fine-tuned Qwen3 models at 8B, 14B, and 32B scales. Results on SWE-bench Verified show a resolved rate of 27.2%—a 540% improvement over the base model and 256% over unfiltered fine-tuning. Beyond raw correctness, the models also improved architectural patch quality and demonstrated strong cross-language generalization, making the approach applicable beyond a single programming language.
- Two judges (ACJ and AQJ) replace costly human experts for architectural evaluation of code patches
- Fine-tuning Qwen3 on 3,360 curated instances achieves 27.2% resolved rate on SWE-bench Verified
- Cross-language generalization and significantly improved architectural patch quality over base models
Why It Matters
Automated architectural evaluation enables scalable training of code LLMs for real-world software engineering.