ProGPO: New training method breaks credit trap for AI agents on long tasks

Training LLM agents on long-horizon tasks with sparse outcome rewards is notoriously difficult. Group-based policy optimization methods compare trajectories or steps within a group, but they often hit a 'credit trap': when all rollouts fail, the outcome reward provides no signal, letting repeated low-effect actions dominate. A new paper from researchers (Kaibing Yang et al.) introduces Progress-conditioned Group Policy Optimization (ProGPO) to break this loop.

ProGPO works by tracking which states an agent has visited for the first time. In groups where all samples receive zero outcome reward, it gives higher relative advantage to trajectories or steps that discover more new states. This rewards exploration even in the absence of success. The team tested ProGPO on ALFWorld and WebShop using Qwen2.5-1.5B and 7B models, and found consistent improvements over standard group-based baselines, with particularly large gains on the hardest tasks. The method is simple, doesn't require extra reward models, and could scale to more complex agentic workflows.