SLPO brings RL to latent reasoning models, outperforming explicit CoT
Latent reasoning gets outcome-reward RL via surrogate policy, scaling test-time compute cheaper.
Latent reasoning models—which carry intermediate computations as continuous vectors rather than discrete language tokens—have shown they can match or even surpass explicit Chain-of-Thought (CoT) reasoning at much shorter horizons. However, they remained stuck in imitation learning because outcome-reward reinforcement learning (RL) requires a tractable per-step likelihood and an adaptive stopping mechanism. SLPO (Surrogate Latent Policy Optimization) solves both problems: it introduces a surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy.
In experiments across continuous and soft thinking settings, SLPO significantly improves Pass@k under parallel sampling—meaning it generates multiple candidate reasoning paths and selects the best. Crucially, the model learns to allocate longer latent computation to harder problems, boosting deterministic accuracy. This opens the door to scaling test-time compute for latent reasoners without the token-by-token decoding cost that makes explicit CoT so expensive, potentially making advanced reasoning more accessible for real-time applications.
- SLPO applies outcome-reward RL to latent reasoning by using a surrogate policy density for credit assignment over continuous latent transitions.
- It introduces a correctness-supervised stopping head that learns variable-horizon reasoning, allocating more compute to harder problems.
- The method improves Pass@k under parallel sampling and achieves higher deterministic accuracy compared to baseline latent reasoners.
Why It Matters
SLPO could make large-scale reasoning cheaper and faster by bringing RL scaling to latent models.