Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models

A research team from institutions including Charité – Universitätsmedizin Berlin has developed a novel AI system for automatically detecting and classifying surgical instrument handovers in operating room videos. The model, detailed in a paper submitted to CVPR 2026, tackles the challenging problem of monitoring complex surgical workflows where frequent occlusions and background clutter make manual tracking difficult. It uses a Vision Transformer (ViT) to extract spatial features from video frames and a unidirectional Long Short-Term Memory (LSTM) network to analyze temporal sequences, creating a unified multi-task framework that jointly predicts handover occurrence and direction.

In experiments on a dataset of kidney transplant procedures, the system demonstrated strong performance, achieving an F1-score of 0.84 for detecting when a handover occurs and a mean F1-score of 0.72 for classifying the direction of the transfer (e.g., surgeon to assistant). This outperformed both a single-task variant and a VideoMamba-based baseline for direction prediction. To address the critical need for trust in medical AI, the researchers employed Layer-CAM (Class Activation Mapping) attribution techniques to visualize which spatial regions of the video—such as hands and instruments—were driving the model's decisions, making its reasoning more interpretable to surgical teams.

The research represents a significant step toward automated surgical workflow analysis. By providing reliable, event-level monitoring of instrument exchanges, the system could help reduce procedural errors, standardize training, and create more efficient operating room environments. The team's focus on interpretability through visualization techniques is particularly important for clinical adoption, as it allows surgeons to understand and verify the AI's conclusions rather than treating them as a black box.