Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment

A research team has developed a novel AI framework that significantly improves the accuracy of assessing cerebral palsy (CP) severity from video. The model, detailed in the arXiv paper "Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment," addresses a key limitation of existing methods. Current approaches typically rely on either raw pose sequences or manually crafted gait features alone, failing to capture both detailed motion patterns and biomechanically relevant information simultaneously.

To bridge this gap, the researchers built a dual-stream architecture. The first stream uses a Spatio-Temporal Graph Convolutional Network (ST-GCN) to model skeleton dynamics. The second stream encodes specific, clinically meaningful gait features—like stride length or joint angles—derived from key body points identified as most important via a Grad-CAM analysis. These two streams are fused using a feature cross-attention mechanism.

The result is a system that achieved a classification accuracy of 70.86% across four levels of CP motor severity (GMFCS levels), outperforming the baseline model by 5.6 percentage points. This fusion not only boosts performance but also enhances interpretability, as the model highlights which body parts and gait metrics are most influential in its assessment. The work demonstrates that combining deep learning with domain-specific clinical knowledge creates more powerful and trustworthy diagnostic tools.

This research paves the way for more accessible and frequent motor function monitoring. By requiring only video input—potentially from a smartphone—it could reduce reliance on expensive, lab-based motion capture systems and specialist visits, enabling remote assessments and more personalized treatment tracking for children with CP.