Offline RL's Dirty Secret: Rewards Missing Not at Random — This New OPE Method Finally Tackles It

In offline reinforcement learning, logged batch data often suffers from missing immediate rewards due to sparse record-keeping or censoring beyond certain values. When rewards are missing not at random (MNAR), standard assumptions of ignorability break down, introducing selection bias even after conditioning on states and actions. A new paper from Ziheng Wei, Annie Qu, and Rui Miao, accepted at ICML 2026, tackles this problem head-on. The authors formalize a reward-dependent propensity model and leverage future states as shadow variables to identify the full-data conditional mean reward. They introduce a bridge function that recovers the conditional mean reward without explicitly modeling the MNAR mechanism, estimating it via a min-max procedure that avoids double sampling.

Building on these identification results, the team proposes an estimator styled after Fitted-Q-Evaluation that propagates the recovered rewards while allowing target policies to condition on past missingness indicators. The approach establishes consistency and finite-sample error bounds. Experiments on simulated environments and the MIMIC-III Sepsis dataset demonstrate strong performance compared to existing off-policy evaluation methods that fail to account for MNAR rewards. This work opens the door to more reliable offline evaluation in healthcare, marketing, and other domains where data collection is irregular or censored.