MALLM framework reveals best decision protocols for multi-agent LLMs

Scaling large language models faces diminishing returns and skyrocketing costs. Multi-agent systems (MAS) offer a compelling alternative by distributing tasks among specialized agents that deliberate and decide collectively. However, the critical component—how agents reach a final answer (the decision protocol)—has been underexplored. A new Master's thesis from the University of Göttingen by Lars Benedikt Kaesberg introduces the MALLM (Multi-Agent LLM) framework to systematically evaluate three decision protocols: voting, consensus, and judge. Unlike prior work that tested limited protocols on narrow datasets, MALLM experiments covered seven diverse benchmarks including knowledge-based ones (MMLU, MMLU-Pro, GPQA) and logic-based ones (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0).

The results reveal that no single protocol dominates. Consensus protocols—where agents iteratively refine a shared answer—excel in knowledge-intensive domains, aligning well with tasks requiring factual agreement. Voting and judge protocols, where agents either cast independent votes or rely on a designated evaluator, prove more effective for logic-based and reasoning tasks. A key finding: increasing response diversity by having agents generate solutions independently before discussion substantially improves decision quality across all protocols. Meanwhile, varying the information access each agent had during the decision process showed minimal impact. These insights provide actionable guidance for engineers building multi-agent LLM applications, suggesting that protocol choice should be tailored to the nature of the task at hand.