UIUC researchers fine-tune Whisper to decode infant audio across households

Infant-centered audio understanding from naturalistic home recordings has long been stymied by noisy, sparse data and dramatic acoustic variation between families. Researchers at UIUC (Xulin Fan, Jialu Li, Mohammad Nur Hossain Khan, Kexin Hu, Bashima Islam, Mark Hasegawa-Johnson, Nancy L. McElwain) present a robust solution that repurposes OpenAI's Whisper as a foundation encoder. Their multi-tier audio tagger combines a LoRA-finetuned Whisper encoder with a lightweight, target-speaker-aware Transformer designed for long-context inference and framewise prediction across tiers. A key innovation is the factorized speaker-token design — a shared tier token plus a learned family-specific offset — which decouples speaker identity from audio semantics and reduces family bias during training. They also introduce a sequence-level smoothing loss to enforce temporal coherence across frames, ensuring predictions don't flicker unpredictably over daylong recordings.

The system elegantly tackles two core problems: low signal-to-noise ratio in real-world infant settings and the scarcity of labeled infant audio data. By conditioning on structured speaker tokens and using LoRA to efficiently adapt Whisper's pre-trained representations, the model achieves generalizable cross-family performance without requiring massive labeled datasets. This enables practical deployment for daylong at-home monitoring, where the audio stream includes everything from crying and babbling to caregiver speech. Accepted to Interspeech 2026, this work demonstrates how large pre-trained speech models can be adapted to highly specialized, low-resource domains. For developmental researchers and pediatric monitoring, this means robust automated tagging of infant vocalizations across heterogeneous households — potentially enabling early screening, parent-child interaction analysis, and longitudinal studies at unprecedented scale with minimal manual annotation.