Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Reinforcement learning (RL) has become central to post-training LLMs for improved reasoning, but the design of rollout pipelines—the trajectories sampled from prompts to termination—is often underreported and ad hoc. A new comprehensive survey by Rohan Surana and 21 co-authors provides an optimizer-agnostic view, formalizing rollout strategies with unified notation. They introduce the GFCR (Generate-Filter-Control-Replay) lifecycle taxonomy, which decomposes rollout pipelines into four modular stages: Generate (proposing candidate trajectories), Filter (constructing intermediate signals via verifiers, judges, critics), Control (allocating compute and making continuation/branching/stopping decisions under budgets), and Replay (retaining and reusing artifacts across rollouts without weight updates, including self-evolving curricula). The framework is complemented by a criterion taxonomy of reliability, coverage, and cost sensitivity to characterize rollout trade-offs.

The survey synthesizes methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. Grounded with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks, the paper also provides a diagnostic index mapping common rollout pathologies to GFCR modules and mitigation levers. This taxonomy aims to help researchers build reproducible, compute-efficient, and trustworthy rollout pipelines, addressing open challenges in scaling RL-based post-training for LLMs.