Most Multi-Agent LLM Coordination Gains Are Within Noise Floor, Study Finds

A new preprint from Kaliyev and Maryanskyy takes a hard look at multi-agent LLM coordination benchmarks, asking a simple but critical question: how much of the reported improvement is actually real? The researchers designed a 'paired noise-floor protocol' that compares two configuration-equivalent protocols (matched via code inspection and SHA-256 byte audit) on the same model and benchmark. Using Claude Haiku 4.5 against tau^2-bench retail, they measured trial-0 disagreement across two seeds of n=100 each. The results are sobering: the cleanest contrast gave a pooled gap of just +5 percentage points (Wilson CI [-2, +12]), not statistically significant. Even the largest single-seed contrast (+18pp) failed to replicate on a second seed (-3pp). After Bonferroni correction, no contrast at either seed or pooled was significant.

The study then maps these findings onto the broader multi-agent landscape: seven of ten recent coordination architectures report headline effects that fall below the observed noise floor (upper CI ~15pp), and one more sits inside the envelope. This means many claimed improvements may be indistinguishable from noise when subjected to a same-model paired replication. To address this, the authors propose 'coordination-active pass^k'—a metric that restricts evaluation to trials where the coordination mechanism is logically active—as a minimum reporting standard. They also provide sample-size targets and runtime hooks, and run measurements on ET-MCP, a task-scoped negative-knowledge store. The paper doesn't offer new coordination methods, but presents a rigorous diagnostic that suggests the field needs much higher replication standards before declaring that one architecture 'beats' another.