Study reveals optimal temperature trade-offs for neural reward models in RL

A new arXiv paper by Higuchi et al. provides a theoretical analysis of how neural reward models learn features when used in KL-regularized policy optimization. The authors study a Gaussian single-index model where the true reward is a function of a hidden direction, and the learner first estimates this direction from reward-weighted samples before fitting a readout layer via ridge regression.

Key finding: exponential reward weighting alters the Hermite signal available to the first layer of the neural network. When the feature-learning temperature exceeds a dimension-free O(1) threshold, a constant fraction of neurons successfully recover the hidden direction—a weak-recovery complexity governed by the generative exponent. The paper then derives bounds on the policy value gap for two weighting schemes: an idealized label-weighted fit using true rewards and a practical surrogate-weighted fit using learned rewards.

These bounds reveal an admissible set of deployment temperatures that balance the benefit of lower temperatures (sharper policies) against the increased learning cost from exponential weighting. The surrogate-weighted case introduces proxy-dependent factors that shrink this admissible set, highlighting a practical constraint. The work provides a rigorous framework for understanding when and why reward model training leads to good downstream policies, with implications for RLHF and related methods.