Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction

A consortium of researchers has developed a state-of-the-art AI system that can accurately parse and extract medical information from the notoriously difficult audio of real-world, bilingual clinical conversations. The system, detailed in a new arXiv paper, tackles the DISPLACE-M dataset, which features Hinglish (Hindi-English) dialogues with rapid turn-taking and highly overlapped speech between doctors and patients. To solve this, the team created a two-part architecture: first, an End-to-End Neural Diarization with Vector Clustering (EEND-VC) model precisely identifies who is speaking (doctor or patient) even during dense overlaps. Second, for transcription, they adapted a Qwen3 Automatic Speech Recognition (ASR) model through domain-specific fine-tuning, Devanagari script normalization, and dialogue-level error correction using a Large Language Model (LLM).

This cascaded, text-based approach achieved a remarkably low transcription error rate (tcpWER) of 18.59% on the challenging dataset. The researchers benchmarked their open system against proprietary, multimodal end-to-end models that process audio directly for information extraction. While the proprietary models set the performance ceiling, the team's open architecture proved highly competitive, ultimately securing first place out of 25 participants in the official DISPLACE-M challenge. All code and models from this winning entry have been made publicly available, providing a significant open-source resource for the speech processing and medical AI communities. The work demonstrates that well-designed, modular open-source systems can achieve top-tier results on complex, real-world problems where data is messy, code-switched, and acoustically challenging.