Autonomous Skeletal Landmark Localization towards Agentic C-Arm Control
Fine-tuned multimodal AI models achieve competitive landmark localization, enabling autonomous adjustments in surgery.
A research team including Jay Jung, Ahmad Arrabi, and Jax Luo has published a paper proposing a novel agentic framework for autonomous C-arm control in surgical settings. The system fine-tunes Multimodal Large Language Models (MLLMs) to perform skeletal landmark localization from X-ray images, a critical step for accurately positioning the C-arm fluoroscopy device during procedures like orthopedic surgery. When conventional Deep Learning models fail, forcing clinicians to manually operate the machine, this MLLM-based approach can incorporate feedback and use reasoning to make iterative adjustments, aiming to reduce delays in time-sensitive interventions.
The researchers trained and evaluated two MLLMs on both an annotated synthetic X-ray dataset and a real X-ray dataset, where each image was paired with specific skeletal landmarks. Quantitative results showed the fine-tuned MLLMs achieved performance competitive with a leading DL benchmark across all localization tasks. Crucially, qualitative experiments demonstrated the system's advanced capabilities: the MLLM could correct an initially incorrect prediction through a reasoning process and could sequentially navigate the C-arm toward a target location, showing evidence of spatial awareness. The code is publicly available, and the work was accepted for IJCARS: IPCAI 2026, marking a significant step toward fully autonomous, reasoning-based surgical assistive systems.
- The framework fine-tunes Multimodal LLMs (MLLMs) to localize skeletal landmarks in X-rays for C-arm control.
- It showed competitive quantitative performance against a leading Deep Learning model on both synthetic and real datasets.
- Qualitative tests proved the MLLM can reason to correct errors and autonomously navigate the C-arm to a target.
Why It Matters
This could automate complex surgical imaging setup, reducing procedure time and clinician workload in emergency interventions.