Why dynamically routing multi-timescale advantages in PPO causes policy collapse (and a simple decoupled fix) [R]

An undergraduate researcher has published a paper and open-source code diagnosing a critical failure mode in advanced reinforcement learning. The problem occurs when training agents with Proximal Policy Optimization (PPO) using dynamically routed advantages from multiple timescales (e.g., γ = 0.5, 0.9, 0.99). Instead of learning optimal behavior, the agent's optimization process gets hacked: the temporal attention mechanism is manipulated to minimize the surrogate loss, ignoring the actual control task. This leads to irreversible policy collapse or agents getting stuck in bizarre local optima, like a LunarLander agent hovering endlessly to hoard small, short-term rewards.

The researcher's analysis pinpointed two core pathologies: 'Surrogate Objective Hacking' and the 'Paradox of Temporal Uncertainty.' The proposed fix, termed 'Target Decoupling,' is a simple architectural change. It advocates for 'Representation over Routing,' where multi-timescale predictions are kept solely on the Critic side to build robust representations, while the Actor is strictly updated using only the long-term advantage signal. This decoupling prevents the optimizer from taking shortcuts. When applied, the LunarLander agent stopped its fearful hovering and learned a highly fuel-efficient, perfect landing, consistently breaking the 200-point benchmark.

To make the findings accessible, the researcher built a strict 4-stage Minimal Reproducible Example (MRE) in pure PyTorch. This standalone project allows others to see the agent crash, hover, and succeed within minutes. The complete code, paper, and demonstration GIFs are fully open-sourced on GitHub and arXiv, providing a clear reference for developers building multi-horizon AI agents and aiming to demystify the underlying math of PPO and temporal credit assignment.