New Theory Reveals Optimal Chain-of-Thought Depth in LLMs

Chain-of-thought (CoT) reasoning has become a standard technique to elicit multi-step reasoning from large language models by generating intermediate steps at inference time. However, until now the scaling behavior of generalization with CoT depth has been poorly understood. Researchers Kaito Takanami and Cengiz Pehlevan address this gap with a theoretically solvable model of CoT for in-context weight prediction in linear regression, representing test-time reasoning as iterative refinement of a weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, they derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length.

Their analysis reveals a sharp phase transition separating regimes of exponential improvement, polynomial improvement, saturation, and overthinking. They characterize how the optimal reasoning depth scales with data and context: deeper reasoning is most effective when both pretraining and in-context information are rich, while limited pretraining or context makes longer chains prone to error amplification or saturation. These predictions are validated through experiments on fully learned linear attention and softmax attention models. The work provides a unified theoretical account of how test-time CoT depth affects generalization, offering practical guidance for choosing reasoning steps in deployed LLMs without wasted compute.