DeepSeek-R1 pairs crush multi-agent coding tests, LLaMA 3.2 and Qwen3 show role alignment
In a controlled test of 12 LLM pairs on a simple Fibonacci coding task, only DeepSeek-R1 teams converged correctly from the start—every other open-source model either showed role alignment but diverged or never converged at all, revealing a deep gap between model capability and multi-agent reliability.
A new study from researchers at Chalmers University of Technology and Ericsson systematically analyzed conversations between two LLM-based agents—a Designer and a Programmer—across 12 model combinations from 7 open-source large language models: Gemma 2, Gemma 3, LLaMA 3.2, LLaMA 3.3, DeepSeek-R1, MiniCPM, and Qwen3. The task: collaboratively develop a Fibonacci game. The team measured three dimensions: efficiency (speed and stability of convergence), consistency (role alignment via BLEU and ROUGE scores), and effectiveness (compilation success and error resolution).
Results revealed a clear winner: the DeepSeek-R1:DeepSeek-R1 pair was the only combination that converged to the correct solution from the very first iteration and sustained it consistently to the final iteration. In contrast, the LLaMA 3.2:LLaMA 3.2 and Qwen3:Qwen3 pairs demonstrated strong Designer:Programmer role alignment but diverged from the correct solution over time. All other pairs deviated early and never recovered. The findings underscore that naively allowing agents to interact does not guarantee stable outcomes—unstructured dynamics can lead to error propagation, premature consensus on incorrect solutions, or endless disagreement. This work is a critical step toward calibrating convergence and stop conditions for future autonomous software engineering.
- DeepSeek-R1 stands out as the only open-source model pair that reliably converges on a simple multi-agent coding task, highlighting the value of reasoning-optimized architectures.
- The majority of open-source model pairs fail to converge, suggesting that multi-agent coordination remains a weak point even as single-agent performance improves.
- The $30 billion market for AI software engineering tools may face adoption delays unless models can demonstrate consistent multi-agent reliability beyond simple benchmarks.
Why It Matters
Multi-agent coordination is the next frontier for autonomous coding, but current open-source models show it remains fragile.