RL + Digital Twins Boost Surgical VideoQA by Decoupling Perception and Reasoning

A new reinforcement learning (RL) framework from Yiqing Shen and colleagues trains large language models (LLMs) to tackle reasoning-intensive surgical video question answering (VideoQA) by operating over digital twin representations. Traditional methods compress video into discrete tokens, fragmenting spatial-temporal relationships and limiting multi-step reasoning. The new approach instead uses surgical foundation models to construct digital twin representations, then applies RL to decouple perception from reasoning. Hierarchical representations capture information at the frame, temporal window, and procedure levels, each with probabilistic uncertainty estimates. A novel reward function combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration.

The team also introduces REAL-Colon-Reason, a colonoscopic benchmark with 2,000 question-answer pairs across three complexity levels. The model achieved state-of-the-art performance not only on this benchmark but also on the existing REAL-Colon-VQA and EndoVis18-VQA benchmarks. The decoupled architecture allows the LLM to reason over the digital twin's structured representations rather than raw video tokens, enabling more robust multi-step reasoning. This work demonstrates that explicitly separating visual understanding from logical inference significantly improves performance on tasks requiring semantic, spatial, and temporal reasoning.