For too long, teaching LLMs to reason has been like forcing a parrot to recite Shakespeare. Trajectory imitation — behavioral cloning — worked in controlled settings but broke down the moment the student drifted. Researchers are finally calling time on that approach.
Two preprints from 2025 and 2026 show the shift. The first, from October 2025, introduces an offline RL framework that uses inverse reinforcement learning to extract robust reward models from LLM trajectories. Instead of copying actions, the model learns what makes a reasoning path good. The second, Teacher-Guided Policy Optimization (TGPO), dated May 13, 2026, tackles the "policy drift" problem head-on: it remains stable even when the student's policy diverges significantly from the teacher. That's a strategy — not a parrot.
Why should you care? Because every chatbot, coding assistant, or reasoning benchmark you've interacted with has been trained on some form of imitation. The next generation won't just replicate examples; they'll adapt to novel problems. It's the difference between a recipe-follower and a chef who understands flavor.
Both ICLR 2026 (starting April 22 in Rio de Janeiro) and ICML 2026 (in Seoul) are packed with reasoning papers. The community isn't debating whether to move beyond imitation — it's competing to build the best method. TGPO and the imitation-enhanced RL framework represent two poles: on-policy distillation versus offline reward learning.
Trends suggest the real breakthrough will come from hybrid methods that blend static demonstrations with dynamic correction. We might see a framework that first learns a reward signal from teacher data, then uses on-policy optimization to stay robust to drift. That combination could finally give us LLMs that reason like experts — not just mimic them.
The race is on. The winners won't be the ones with the biggest models, but those who teach their models to *learn* how to reason. And that starts with ditching trajectory imitation for good.