Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity

Contextual Markov decision processes (MDPs) are widely used in fields ranging from biostatistics to machine learning, but existing offline methods often fail when the underlying environment is non-stationary or irregular. A new paper by Bhattacharyya, Chakrabarty, and Banerjee introduces the first theoretically guaranteed estimator for offline contextual MDPs that works without assuming stationarity. Their key innovation leverages T-estimation, a powerful statistical technique from Baraud (2011), to overcome the endogenous challenges of non-stationarity and model irregularity. The estimator achieves oracle risk bounds under two distinct loss functions and provides finite-sample guarantees for optimal control policies using only historical data.

The work is grounded in complete generality, meaning it does not rely on restrictive assumptions that plague prior approaches. The authors first design a procedure to select an estimator from a sample of a contextual MDP, deriving bounds that match the best possible performance (oracle risk). They then apply this density estimate to determine optimal control, with finite-sample guarantees for the cost function. Published in the Transactions on Machine Learning Research (TMLR), the 28-page paper marks a significant step toward reliable offline reinforcement learning in real-world scenarios where environments drift or data is limited. This opens doors for safer deployment in healthcare, robotics, and personalized recommendation systems.