LLMZero uses LLM agents to auto-discover RL training strategies

A new paper from Haoyang Fang and colleagues introduces LLMZero, a system that leverages LLM agents to automatically discover adaptive training strategies for reinforcement learning (RL) post-training. Instead of relying on fixed schedules, LLMZero uses tree search to explore training trajectories, diagnosing pathologies at each checkpoint and proposing coordinated multi-parameter transitions. This approach addresses a fundamental limitation of existing RL post-training: fixed schedules cannot express the non-stationary exploration-exploitation tradeoffs that regularization must track. The system uncovers a recurring empirical pattern—capacity parameters (e.g., model size) accumulate monotonically across stages, while regularization parameters (e.g., entropy coefficients) oscillate in response to shifting training dynamics.LLMZero was tested across 4 diverse GRPO (Group Relative Policy Optimization) tasks, demonstrating significant improvements: 9% to 140% relative gain over the base model and 6% to 15% over traditional grid search. It consistently outperformed random search and skill-based agent baselines. Crucially, the discovered structural principle transfers across tasks—meaning strategies take different forms but share similar parameter dynamics. This provides actionable design rules for multi-stage training, potentially automating a process that currently requires extensive human trial-and-error.