Cross-lingual AI detects dementia from speech without transcription

A new paper from researchers Chukwuemeka Ugwu and Oluwafemi Richard Oyeleke introduces an ASR-agnostic multimodal spectrotemporal modeling approach for early dementia detection. Unlike traditional systems that rely on automatic speech recognition (ASR) transcripts, this method extracts spectrotemporal displacement fields from consecutive Mel spectrogram frames, capturing shifting spectral energy patterns as digital biomarkers of cognitive decline. These features are fused with CNN-ConvGRU acoustic embeddings via a learned cross-attention mechanism and aggregated using a Transformer encoder with learnable query pooling. A composite temporal loss enforces smoothness and contrastive coherence across segments. The framework was validated on three languages: English DementiaBank, Slovak EWA-DB, and Spanish Ivanova.

Results reveal stark corpus-dependent behavior. The Slovak model achieved 83.9% accuracy with full multimodal fusion, but a unimodal audio encoder alone reached 93.7%, suggesting the visual modality (spectrogram) already contains all necessary signal. In contrast, removing cross-attention collapsed Spanish performance to 53.7%, indicating fusion is essential when signal is distributed across modalities. The English baseline returned only 53.2%, confirming known recording artifacts bias earlier benchmarks. Auxiliary temporal losses converged to language-invariant values, hinting at architectural stability across languages. The work underscores that multimodal fusion strategies must be tuned per corpus—essential in some cases, counterproductive in others, and irrelevant when no signal exists.