PATR Boosts RL for LLM Agents by 9.3 Points via Tree Rollout

Reinforcement learning for LLM agents traditionally relies on methods like GRPO and RLOO, which sample multiple independent complete trajectories for advantage estimation. In long-horizon agentic tasks, this uniform rollout often wastes resources on dead-end attempts while under-exploring promising intermediate states. The multi-turn structure of agent trajectories naturally forms a tree, where each turn is a decision point. This insight reframes effective exploration as deciding where to branch.

To address this, the authors propose Process-Scorer Guided Adaptive Tree Rollout (PATR). PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration. Evaluated on FrozenLake and SWE-Bench (largely unexplored by prior tree-rollout agent RL methods), PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, demonstrating that process-guided tree rollouts are a scalable strategy for multi-turn RL.