New study reveals why RL-trained models beat SFT on math reasoning

A new study from Rahman, Gurugubelli, Ankit, Zhu, and Balwani investigates why reinforcement learning (RL) fine-tuned models outperform supervised fine-tuned (SFT) models on mathematical reasoning. Using linear probes trained on layer-wise hidden states, the researchers found that RL models achieve significantly higher accuracy in predicting answer correctness, indicating more linearly separable and structured internal representations. Mean ablation studies further revealed that RL fine-tuning creates a hierarchical architecture: deeper layers become progressively more critical for reasoning, while SFT models distribute importance uniformly across layers. This fundamental restructuring explains RL's superior performance on complex math problems.

The study also examined token-count variability under repeated sampling to assess adaptive compute allocation. While some RL-tuned models showed higher variability compared to SFT counterparts, others exhibited strong consistency, suggesting that token allocation patterns depend more on the overall training pipeline—including data, reward design, and hyperparameters—than on RL versus SFT alone. The authors interpret this variability as revealing the spread of plausible on-policy reasoning, distinguishing models with stable policies from those with under-determined, potentially non-identifiable solution behavior. Presented at the XAI4Science workshop (AAAI 2026), the work provides mechanistic insight into why RL training restructures model representations for mathematical problem-solving.