New Study: Sign Language AI Models Learn Phonology But With Architectural Trade-offs

A new study published on arXiv and accepted to CogSci 2026 investigates whether deep learning models for Sign Language Recognition (SLR) truly understand the phonological building blocks of sign languages—such as handshape, location, and movement—or merely exploit low-level statistical correlations. Led by Kayo Yin and colleagues, the research probes models trained on American Sign Language (ASL) using minimal pairs (sign pairs differing in one phonological parameter) and compares their internal representations to human perceptual similarity judgments.

The results reveal clear architectural trade-offs: pose-based models (using skeletal keypoints) show strong sensitivity to handshape contrasts, while pixel-based models (raw video input) better capture location changes. Notably, pose-based models achieve a correlation of r≈0.49 with human judgments, indicating limited but emergent phonological sensitivity. However, the authors conclude that current training paradigms fail to overcome inductive biases inherent to each architecture, suggesting that future SLR systems must integrate multiple input modalities and more phonologically-aware training objectives to reach human-level perception.