Researchers propose M^3QAFrame for multi-modal medical QA
New AI reads text and images to answer complex medical queries across multiple sections.
A team of researchers (Anisha Saha, Vaibhav Rathore, Abhisek Tiwari, Akash Ghosh, Sai Ruthvik Edara, and Sriparna Saha) has introduced M^3QAFrame, a novel framework for multi-modal multi-span medical question answering. The system addresses a critical gap in existing MedQA systems: they handle only text, while real medical documents often include images (X-rays, diagrams, charts). M^3QAFrame takes a context document, a natural language query, and accompanying images as input. It processes text and image embeddings through a transformer-based architecture to determine sentence and image relevance, then outputs an answer that includes both extracted textual spans and relevant visual content.
To support this work, the authors curated the M^3 QuestionIng dataset, featuring medical contexts with associated images, extractive answers spanning multiple text and image regions, plus query intent and type labels for better comprehension. In extensive experiments, M^3QAFrame consistently outperformed existing methods across multiple evaluation metrics, demonstrating the power of combining vision and language for complex medical queries. This work could significantly improve AI-assisted diagnosis and medical literature review by providing more complete, visual-annotated answers.
- M^3QAFrame uses a transformer architecture to jointly process text and image embeddings for multi-span answer selection.
- The M^3 QuestionIng dataset includes intent and query type labels for improved comprehension of medical queries.
- Framework beats existing methods across multiple evaluation metrics, addressing the real-world gap of multi-modal documents.
Why It Matters
Brings medical AI closer to real clinical scenarios by combining text and images into comprehensive, context-aware answers.