Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation

A research team has published a novel AI framework on arXiv that could transform how aphasia, an acquired language disorder, is assessed. The paper, 'Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation,' proposes moving beyond traditional tools like the Western Aphasia Battery-Revised (WAB-R), which measures isolated skills, by using AI to analyze spontaneous discourse as a more holistic measure of everyday language ability. The core innovation is the shift from analyzing isolated linguistic or acoustic features to modeling the complex, structured interaction between a patient's speech and their gestures.

The team's framework represents each patient's discourse as a directed multi-modal graph. In this graph, nodes are lexical items (words) and gestures, while edges encode the transitions between them—word-to-word, gesture-to-word, and word-to-gesture. They employ a GraphSAGE neural network to learn embeddings that capture information from both a node's immediate neighbors and the overall graph structure. The key finding is that aphasia severity is encoded not in isolated word distribution but in these structured multimodal interactions. This architecture offers a path toward reliable, automated assessment that could be deployed for efficient bedside screening or remote telehealth monitoring, providing a more nuanced and continuous evaluation tool for speech-language pathologists.