Whisper + LLM framework detects dementia with 90.14% accuracy

A new research paper—accepted at INTERSPEECH 2026—introduces a multimodal framework for early dementia detection by combining acoustic embeddings from Whisper with LLM-augmented linguistic analysis. The system uses Whisper's encoder outputs as acoustic representations, processed through temporal networks with attention pooling to create fixed-dimensional embeddings. Meanwhile, the ASR transcripts from Whisper are fed into a large language model (the paper doesn't specify which one, but likely a GPT-series or LLaMA) to extract interpretable linguistic biomarkers including lexical diversity, syntactic complexity, semantic coherence, and discourse patterns. A gated fusion network then intelligently integrates both acoustic and linguistic modalities.

On the ADReSS and ADReSSo benchmark datasets, the method achieved F1-scores of 89.47% and 90.14% respectively—significantly outperforming single-modality baselines. Ablation studies confirmed that the multimodal approach consistently yields better results than either the acoustic-only or language-only pipeline. This work is a strong step toward non-invasive, scalable dementia screening: simply recording a patient's speech and running it through the pipeline could flag early cognitive decline, no need for expensive brain scans or invasive tests. The open-source Whisper model and widely available LLMs make this approach accessible to healthcare providers and researchers alike.