NTU study: Speech LMs fail at human-like sound symbolism perception

Sound symbolism—the human tendency to associate speech sounds with perceptual qualities such as roundness or sharpness—is driven primarily by acoustic properties rather than spelling. A new study from National Taiwan University (NTU) researchers, submitted to SLT 2026, investigates whether Speech Language Models (SLMs) exhibit this same intuition. Unlike prior evaluations that relied on text or images, the team used genuine human speech recordings, comparing model judgments against human data across auditory, crossmodal, and visual components of the effect.

The results are striking: SLMs' auditory judgments align poorly with human perception and miss the acoustic cues—especially spectral tilt—that drive human intuitions. Open-weight models cannot reliably link a heard sound to its corresponding shape. A visual-only control rule out shape perception deficits, localizing the weakness to how speech is represented. The study concludes that achieving perceptual alignment in SLMs requires better speech representations that capture the subtle acoustic cues humans naturally hear, not simply stronger visual modules.