DPO achieves 93.5% GSM8K accuracy via nearly orthogonal weight updates

A new paper from Nikolich et al. (accepted at ICML 2026 workshop) systematically dissects the weight-space geometry of six offline reasoning training methods—SFT, RFT, DFT, RIFT, Offline GRPO, and DPO—using Qwen3-4B as the base model with attention-only LoRA. The authors find that SFT, RFT, and RIFT produce nearly colinear weight deltas (cosine similarity ≥0.97, median principal angle ~7° across 144 modules) and yield nearly identical GSM8K accuracy (~87-88%, pairwise McNemar p ≥0.15). In contrast, DPO (Direct Preference Optimization) occupies a near-orthogonal subspace: cosine similarities with SFT are near zero, late-layer CKA collapses to ~0.46, and there is a clear mode-connectivity barrier. Despite using a 10× smaller learning rate per standard convention, DPO achieves the highest accuracy—93.5% on GSM8K (McNemar p<10⁻⁹ vs. all other methods) and 30.0% on AIME26 (vs. 3.3-10.0% for others). Offline GRPO also adds a significant orthogonal component (~67% globally, up to ~86% in late layers) but stays in the SFT loss basin and underperforms DPO. The authors note that the update-norm and accuracy gaps reflect joint effects of loss function and optimizer choices; they leave a learning-rate-matched DPO comparison for future work. DFT diverges more in direction than any reward-weighted method despite using the same data.

The practical implication is clear: DPO's unique weight geometry yields substantially better reasoning generalization—at least for math tasks—even with a smaller learning rate. This finding challenges the common intuition that preference-based methods simply fine-tune along the same directions as supervised approaches. The near-orthogonal subspace suggests DPO is learning a fundamentally different representation for reasoning, which may explain its superior out-of-distribution performance on harder benchmarks like AIME26. For practitioners building reasoning models, this study makes a strong case for preferring DPO over alternatives like RFT or Offline GRPO, especially when aiming for high accuracy on both standard and challenging math problems.