Peer Discussion Beats Hierarchical AI Review for Math Reasoning

A new study by Chih-Hsuan Yang and nine others challenges a core assumption in multi-agent AI systems: that dedicated reviewer roles reliably improve reasoning. Using 4,181 Omni-MATH problems with OpenAI's gpt-oss-120b model, they compared a planner-executor-reviewer (PER) pipeline against broadcast-style peer discussion where agents freely exchange critiques. On easier problems, both approaches performed similarly, but from tier 4 onward, peer discussion pulled ahead sharply. Surprisingly, PER’s reviewer was more precise at spotting errors (0.861 vs 0.644), yet its critiques were far less likely to be acted upon by the solver agent. This “uncoupling” of reviewer quality from critique uptake means a system can detect errors brilliantly yet fail to fix them.

The team dug into why critiques fail to translate into better answers. In PER, forcing the solver to explicitly acknowledge the reviewer’s feedback actually lowered final accuracy, suggesting that mandatory acknowledgment may interfere with reasoning. Instead, when the reviewer’s guidance was embedded directly into the solver’s working context (e.g., hints woven into the problem-solving scratchpad), follow-through improved but still didn’t close the gap to broadcast. The authors conclude that reviewer-centric evaluation can overstate system quality—a protocol may spot errors well yet still solve fewer problems if it cannot ensure that those critiques change the final output. This has direct implications for building robust multi-agent systems in science, math, and any domain requiring iterative revision.