SGPO: New training method boosts LLM reasoning by 2.2 points

A new paper from a team of researchers introduces Strategy-Guided Policy Optimization (SGPO), a method that goes beyond standard trajectory imitation to teach language models how to reason. Instead of having a weak model copy the exact solution steps from a strong model (which encourages memorization), SGPO extracts structured strategy descriptions from the strong model's responses. For each problem, the weak model generates both an autonomous trajectory and a strategy-guided trajectory, enabling direct comparison of behavior with and without guidance. The framework uses a token-level forward-KL divergence objective to selectively transfer the distributional shift introduced by strategy conditioning into the unguided policy, with proximal constraints for stability. An adaptive weighting mechanism strengthens guidance when autonomous exploration fails and reduces it as the model's own competence grows.

Experiments on four mathematical benchmarks across two model families (Qwen2.5 and Llama) show SGPO consistently outperforms supervised fine-tuning, on-policy reinforcement learning, and hybrid-policy baselines. The average score on Qwen2.5-7B-Instruct improves by 2.2 points over the strongest baseline. Analysis reveals the forward-KL objective provides an inherently selective distillation signal that beats direct trajectory imitation, and strategy distillation exhibits complementary scaling with base model capability. The work suggests that teaching models to understand reasoning strategies, rather than parrot specific solutions, leads to more transferable problem-solving skills.