Offline RL for Adaptive Policy Retrieval in Prior Authorization

A research team has published a paper on arXiv detailing a novel AI system that applies offline reinforcement learning (RL) to automate and optimize the prior authorization (PA) process in healthcare. The system, developed by Ruslan Sharifullin, Maxim Gorshkov, and Hannah Clay, models the task of retrieving complex, fragmented insurance coverage policies as a Markov Decision Process (MDP). Instead of using a static 'top-K' retrieval strategy that fetches a fixed number of policy sections, their AI agent learns to iteratively select relevant policy chunks or decide to stop and issue a coverage decision, dynamically balancing accuracy against the cost of additional lookups.

The researchers trained policies using Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Direct Preference Optimization (DPO) on logged data from synthetic PA requests derived from public CMS coverage data. On a test corpus of 186 policy chunks spanning 10 procedures, CQL achieved a high 92% decision accuracy but did so via exhaustive retrieval. The standout was the DPO-trained policy, which matched that 92% accuracy while using only 10.6 retrieval steps on average—a 47% reduction from the baseline's 20 steps. This placed DPO in a 'selective-accurate' region on the performance frontier, dominating other methods.

The work demonstrates that advantage-weighted or preference-based policy extraction, as used in DPO, is crucial for learning an efficient, selective retrieval strategy. An ablation study on step costs showed that only at a higher cost penalty (λ = 0.2) did the CQL policy transition from exhaustive to selective retrieval, highlighting the importance of the reward function's design. The system represents a significant step toward AI agents that can navigate complex documentation with human-like efficiency, reducing administrative burden.