Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

A research team led by Zhuoxu Huang has introduced CalibRL, a novel framework designed to solve a critical problem in training advanced multi-modal AI models. When using Reinforcement Learning with Verifiable Rewards (RLVR) to enhance the reasoning of models like GPT-4V or Claude 3, the vast state space and sparse rewards often cause the training process to collapse. The model either stops exploring new solutions (entropy collapse), degrades in performance, or gets stuck exploiting suboptimal behaviors. CalibRL addresses this by enabling 'controllable exploration,' a hybrid approach that guides the AI's learning process more effectively than random sampling.

The technical innovation lies in two core mechanisms. First, a distribution-aware advantage weighting scales policy updates based on how rare a successful action is, which helps preserve diversity in exploration. Second, an asymmetric LeakyReLU activation function uses expert knowledge as a calibration baseline to prevent overconfident updates while maintaining their corrective direction. This hybrid-policy approach reduces the mismatch between the model's learned behavior and expert demonstrations. Tested across eight diverse benchmarks, CalibRL demonstrated consistent performance improvements, offering a more stable and efficient path to training sophisticated multi-modal reasoning agents that can process both text and visual information.