New study shows SSL speech models flip performance in MCI detection

A new preprint from researchers at multiple institutions (Kopar et al.) explores how speech representations relate to cognitive assessment hierarchies in mild cognitive impairment. Using 5,754 German neuropsychological assessment recordings, the team evaluated six cognitive tasks across three score levels: task, domain, and global. They compared traditional hand-crafted acoustic features (e.g., pitch, jitter, shimmer) against modern self-supervised learning (SSL) embeddings (like those from wav2vec 2.0 or HuBERT).

Results showed a non-monotonic relationship: SSL representations generally outperformed hand-crafted features at lower hierarchical levels, but this advantage reversed when classifying MCI—hand-crafted features actually performed better for the top-level diagnosis. Additionally, task-specific constraints created a trade-off. Tasks with more response freedom (e.g., describing a picture) exhibited performance dilution as the hierarchy ascended, behaving like "specialist" representations. Conversely, highly structured tasks (e.g., word list recall) improved with higher-level aggregation, acting as "generalist" representations. These findings highlight important design considerations for building reliable AI systems for automated clinical speech analysis, especially for dementia screening.