New LURE estimator solves offline RL with hidden actions

Standard offline reinforcement learning (RL) assumes actions in the training dataset are observed without error. But in real-world applications—like healthcare, robotics, or finance—true actions are often hidden, with only noisy proxies available. This leads to biased and unreliable policy evaluations. A new paper by Zeyu Bian, Ying Zhou, and Yifan Cui tackles this gap head-on with LURE (Learning from the Unseen: Robust Estimator), the first offline RL method designed for hidden actions.

LURE leverages the next-state variable as a natural proxy for unobserved actions, establishing identification of the policy value. The estimator is multiply robust—remaining consistent under several combinations of correctly specified nuisance components—and is asymptotically normal for valid statistical inference. The authors validate LURE through simulations and a sepsis management application using the MIMIC-III database, demonstrating practical effectiveness where action records are often incomplete or noisy.