RLVR Research Reveals Sparse, Low-Rank Updates Reshaping LLM Training

A new synthesis of papers on RLVR (Reinforcement Learning with Verifiable Rewards) reveals that its updates differ qualitatively from supervised fine-tuning (SFT). Key weight-level studies show RLVR updates are approximately 80% sparse, compared to SFT's 20%, and rotate the principal subspaces of the model's Hessian/eNTK eigenvectors by only about 5 degrees—versus 50 degrees for SFT. This suggests RLVR operates in a much smaller, more constrained subspace. One paper even finds that rank-1 LoRA is essentially equivalent to full policy-gradient RL, and that RLVR updates are so consistent they can be predicted from early checkpoints.

Behavioral results add further nuance. RLVR appears to shift models along a bias-variance tradeoff: given many attempts, RLVR-tuned models are more consistently correct, but the base model gets a higher percentage of problems correct at least once. This implies RLVR mostly elicits existing capabilities rather than creating new ones. Other papers show RLVR causes less task forgetting than SFT, and that even random rewards can boost Qwen math performance—likely by suppressing low-probability behaviors. The author's mental model frames these changes as shifts in 'propensity' (singular values) rather than 'lability' (eigenvector rotation).