New study shows Chain-of-Thought isn't always better in LLMs

A new peer-reviewed arXiv paper by Tughanbulut Kurtulush systematically dismantles the myth that Chain-of-Thought (CoT) prompting universally enhances LLM reasoning.

The research introduces a conceptual framework based on the H_dp bandwidth bound, arguing that CoT acts as a bypass for serial computation that exceeds a transformer's single-pass capacity. Testing three instruction-tuned models (Qwen-2.5-7B/32B, Llama-3.1-8B) across five benchmarks (GSM8K, MATH, MMLU, ARC, HumanEval), the study found dramatic performance variations. On high-depth P-complete tasks like GSM8K and MATH, CoT delivered a remarkable +54 to +68 percentage point recovery gap across all models. Meanwhile, on shallow TC^0 tasks like MMLU and ARC, CoT showed minimal impact (Delta in [0.0, +4.6] pp) or even slight degradation, though high baseline accuracies raised concerns about data contamination. The intermediate HumanEval tasks revealed model-size-dependent effects, ranging from +23.2pp (32B) to -28.7pp (7B). The research pre-registered on OSF with full code and data available, suggesting CoT is not a universal reasoning enhancer but rather a targeted solution for tasks straining single-pass capacity.