SpectCount uses synthetic audio to fix LALMs' spectrotemporal blind spots

Large audio language models (LALMs) extend LLMs with audio encoders, but their scaling is bottlenecked by a scarcity of high-quality annotated audio data. By probing signal detectability, researchers from Seoul National University (Kim et al.) identified fine-grained spectrotemporal perceptual weaknesses in a foundation LALM. To address this, they propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach that relies entirely on synthetic audio signals generated on-the-fly—no real-world audio, annotations, or pretrained generative models are used. This targeted synthetic signal method resolves the identified weaknesses and, surprisingly, improves performance on diverse auditory benchmarks (sound, music, speech) that were not seen during fine-tuning.

The results suggest that weakness-targeted synthetic signals provide a cost-effective path to enhanced auditory understanding in LALMs, potentially reducing dependency on expensive real-world annotations. The approach is fully controllable and scalable, enabling precise improvement of specific perceptual capabilities. This work opens the door to synthetic data strategies for audio foundation models, analogous to recent successes in vision and language domains.