MedAgentBench v3 exposes RL limitations for clinical AI agents

A new arXiv paper from researchers at an undisclosed institution (Mantravadi, Rajgarhia, Desikan, Mukherji) takes a hard look at using reinforcement learning (RL) for clinical protocol execution in FHIR (Fast Healthcare Interoperability Resources) environments. The team audited existing MedAgentBench v1/v2 benchmarks and discovered a glaring issue: a 41.7% 'silent-finish' ceiling—meaning many tasks ended without actionable feedback, making inaction the dominant RL strategy. To fix this, they constructed MedAgentBench-v3 (MAB-v3), a cleaner benchmark with 508 tasks and a much lower 8.9% ceiling.

Training Qwen3-8B on MAB-v3 revealed two fundamental barriers to RL effectiveness. First, a capability ceiling: 10 out of 20 task types had 0% base performance, providing zero gradient for RL to improve. Second, a format-knowledge barrier: 3 of 20 task types required exact clinical codes that are simply undiscoverable through exploration. The result? Pure RL achieved only 18.2% pass@1 accuracy, while rule-based supervised fine-tuning (SFT) reached 34.1%—a 15.9 percentage point gap. The paper proposes a taxonomy (decision/format-knowledge/lookup) to predict RL learnability and prescribes a hybrid approach: use SFT to inject required codes, then RL to learn conditional decision-making.