LLMs suffer diversity collapse when fine-tuned on reasoning tasks

A new paper from MIT researchers Junyi Sha, Renfei Tan, and David Simchi-Levi examines a hidden cost of fine-tuning large language models: the collapse of behavioral diversity. Using controlled tic-tac-toe variants where optimal moves are exactly known, the team measured how standard supervised fine-tuning (SFT) and reasoning-mode generation affect the variety of actions an LLM selects. They found that reasoning modes often reduce the spread of chosen actions without consistently improving decision accuracy. Worse, standard SFT induces a "premature diversity collapse" that goes beyond what the accuracy-diversity tradeoff would require.

The study offers a practical fix: action augmentation — training on all optimal actions per state instead of a single demonstrated move — partially restores diversity. The findings suggest that narrow-support imitation (copying only one correct answer) is a root cause of policy collapse in LLM decision-making. For developers, this means preserving action support during SFT is critical for maintaining exploratory behavior, especially in sequential tasks like game playing, autonomous navigation, or tool use. The paper is available on arXiv under ID 2607.19523.