W2SPO: Weak auxiliary branches boost LLM reasoning 3.5x faster
Just 8 tokens from a weaker model can dramatically improve reasoning.
A new paper from Dayu Wang et al. tackles a fundamental limitation in reinforcement learning for LLMs: self-generated rollouts often fall into repetitive 'reasoning basins'—semantically similar error patterns that provide little policy contrast. The proposed method, W2SPO (Weak-to-Strong Off-Policy RL), uses a weaker but computationally efficient auxiliary model to inject short reasoning segments—as few as 8 tokens—into intermediate steps of the target model's trajectories.
Policy updates are applied only to those inserted segments based on final verifiable rewards, leaving the rest of the trajectory frozen. This localizes exploration to novel states, breaking out of redundant reasoning patterns. Evaluated on mathematical reasoning benchmarks with 4B-parameter models, W2SPO outperforms traditional GRPO under the same sampling budget: Pass@1 rises from 62.3% to 64.2%, while training speed increases by 3.55× because the weak model's segments are cheap to generate and only short spans are updated.
The paper demonstrates that weak auxiliary branches can serve as efficient 'exploration engines' for stronger policies, suggesting a paradigm where lightweight models guide larger ones without full retraining. This opens practical avenues for scaling reasoning capabilities in LLMs with minimal computational overhead.
- W2SPO injects only 8 tokens from a weaker model into target model trajectories to expand exploration support.
- Achieves 64.2% Pass@1 on math benchmarks (up from 62.3%) while training 3.55× faster than vanilla GRPO.
- Method overcomes the 'reasoning basin' problem where self-generated samples lack reward contrast.
Why It Matters
A practical, cost-effective way to boost LLM reasoning accuracy without expensive full-model retraining.