Output Format Choice Can Make or Break AI Coding Agents
Your choice of diff format might be killing 94% of coding agent success.
A controlled experiment from arXiv (July 2026) reveals that output format is not a neutral implementation detail — it can completely flip model rankings and determine success or failure of single-round coding agents. Yang Yang tested three popular models (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) across three output formats (full file, JSON Patch, unified diff) on six tasks from four open-source projects (tqdm, dotenv, requests, jsoup), with 20 repetitions each for a total of 4,013 runs. Remarkably, only tqdm yielded non-zero success rates; the other three projects produced exactly zero successes in 2,551 runs.
The central finding is a strong format × model interaction: Doubao achieves 94% success with JSON Patch (Cohen's h = 1.57, p < 0.001), DeepSeek V4 excels at unified diff (66%, h = 0.63), and Qwen 3.7 Max shows a small but significant preference for full-file output (50%, h = 0.29, p < 0.05). Beyond headline numbers, the paper identifies a distinct failure mechanism — "format misuse" — where agents correctly diagnose a problem but execute it with excessive scope, e.g., replacing an entire file when a one-line fix would suffice. The authors propose a model-specific output strategy and a tool-design principle that constrains format semantics to the agent's own localization step.
- No universally optimal output format: format reorders model rankings and can amplify or suppress differences.
- Doubao 2.0 Pro achieves 94% success with JSON Patch; DeepSeek V4 hits 66% with unified diff; Qwen 3.7 Max slightly prefers full file at 50%.
- Identified 'format misuse' failure: agents apply excessive scope (e.g., full-file replacement for a one-line fix), especially when output format doesn't match localization capability.
Why It Matters
For AI coding tools, choosing the wrong output format per model can cause total failure — personalization is essential.