OR3: New AI retrieves OR video clips by reasoning over actions

Researchers from Johns Hopkins University have introduced OR3, a novel text-to-video retrieval system designed specifically for operating room (OR) footage. Traditional retrieval methods fail at understanding implicit queries, such as 'the step right before clipping,' because they rely on global embeddings that cannot reason over temporal actions. OR3 addresses this by first converting video clips into action-driven digital twins (ActDTs) – structured representations that group concurrent subject-action-object triplets under non-overlapping temporal intervals.

Rather than cross-modal matching, OR3 uses an LLM to generate hypothetical ActDTs from the text query, then performs intra-modal matching with a single encoder trained on ActDT-tailored hard negatives. An evidence-grounded refinement step further adjusts the imagined ActDTs based on discrepancies with top candidates. This allows the system to capture procedure-specific patterns not present in the training data.

Evaluated on a benchmark of 276 implicit queries across four reasoning categories from the MM-OR dataset (386 clips from robotic knee surgeries), OR3 achieved 57.6% recall at rank 1 and 77.3% at rank 5, significantly outperforming the strongest baseline. The work demonstrates that reasoning over temporal action structures, rather than visual similarity alone, is key for effective OR video retrieval.