Researchers find 4.17 Hz speech token rate best for preserving LLM reasoning

Spoken dialogue models typically start from a text LLM backbone, but their reasoning degrades when conditioned on speech instead of text. A new paper by Zhen Ye, Xu Tan, and collaborators attributes this modality gap to a temporal-granularity mismatch: speech tokens are temporally redundant and significantly longer than text tokens for the same semantic content. This dilutes per-token semantic density and disrupts the reasoning dynamics the LLM learned from text. To address this, the authors treat speech token design as a representation selection problem, sweeping frame rates under a frozen LLM backbone with a fixed information rate.

To enable very low frame rates, they introduce factorized FSQ (finite scalar quantization) and a lightweight non-autoregressive audio LM head that scales capacity to nearly 300 bits per frame without sacrificing prediction efficiency. With that bottleneck removed, they test frame rates from 50 Hz down to 2.08 Hz and vary alignment depth. The consistent best regime for speech QA is 4.17 Hz with intermediate-layer representation alignment—a finding that could guide future spoken AI systems. The work is accepted as a long paper at Interspeech 2026 and is available on arXiv.