SARA cuts RLVR rollout costs by 22% with smart early stopping

A new paper from Pixel Nomand, Elena Voss, Marcus Hale, and Sofia Reyes introduces SARA (Sequential Adaptive Rollout Allocation), a method to drastically reduce the compute cost of reinforcement learning with verifiable rewards (RLVR). The core insight is that many prompts quickly become saturated—either all responses are correct or all are incorrect—providing no policy-gradient signal. Existing approaches either oversample and discard saturated prompts (dynamic sampling, DPS) or predict difficulty upfront, both of which waste compute.

SARA reframes per-step rollout collection as a budget-constrained sequential allocation problem. It maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and uses a two-threshold SPRT-style rule to commit effective groups early or abandon saturated ones after a short probe. The freed budget is reallocated to fresh prompts without extra prediction rollouts. On mathematical reasoning and planning tasks with 1.5B and 3B models on a single GPU, SARA matches DPS accuracy while using 22% fewer rollouts. Composing SARA with DPS yields the best accuracy, even slightly above DPS, at 67% fewer rollouts—near-uniform cost. The authors also prove abandonment reliability and expected rollout savings.