When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

A new research paper by Artem Maryanskyy titled 'When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines' resolves a key contradiction in AI team design. While previous studies showed both benefits and drawbacks to using diverse AI agents, this research identifies a 'selection bottleneck' - a crossover threshold in aggregation quality that determines whether diversity helps or hurts. The study proposes a closed-form mathematical threshold (Proposition 1) that separates these regimes, providing a theoretical framework for understanding when to use heterogeneous versus homogeneous AI teams.

In practical experiments spanning 42 tasks across 7 categories (N=210), the research revealed dramatic differences in performance. Diverse teams using judge-based selection achieved an impressive 81% win rate against single-model baselines, with a Glass's Δ effect size of 2.07 indicating strong practical significance. Meanwhile, homogeneous teams using synthesis-based aggregation scored only 51.2% - essentially chance performance. Most strikingly, judge-based selection outperformed Mixture-of-Agents (MoA) style synthesis by ΔWR = +0.631, with synthesis approaches failing to beat the baseline in any of the 42 tasks.

The findings suggest that including a weaker model can actually improve performance while reducing costs (p < 10^-4), challenging conventional wisdom about AI team composition. The research concludes that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines, offering practical guidance for developers building multi-agent AI systems.