Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM

A research team from Virginia Tech has published a groundbreaking study on AI-powered adaptive captioning, moving beyond simple text transcription to include vital emotional and non-verbal information. The paper, "Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM," presents four distinct prototypes that embed cues like speaker facial expressions, body gestures, keyword highlighting, and emojis directly into the caption stream. This addresses a critical gap in traditional accessibility tools, which often strip away the emotional context and emphasis crucial for understanding complex STEM material.

In a pilot and main study involving 24 participants from the Deaf and Hard of Hearing (DHH) and neurodivergent communities (including those with ADHD), the researchers found measurable benefits. Specific prototypes led to reduced self-reported cognitive load and improved comprehension scores compared to standard captions. Crucially, the study highlights the need for customization; user preferences varied significantly, especially regarding the use of emojis, pointing to a future where accessible tech adapts to individual neurodivergent profiles rather than offering a one-size-fits-all solution.

The work, accepted at the Affective Computing and Intelligent Interaction (ACII) Conference 2025, bridges human-computer interaction (HCI) and affective computing. It provides a concrete design framework for integrating emotional intelligence into real-time assistive technologies. The findings challenge the industry standard for captions and set a new benchmark for creating educational content that is truly inclusive for learners with diverse cognitive and sensory processing needs.