Mel-LLM: No speech encoder needed, LLM reads spectrograms directly

A new paper by Ruchao Fan and colleagues challenges the standard architecture for speech-aware LLMs, which typically rely on a pre-trained speech encoder to convert audio into semantic-rich representations. Instead, they introduce Mel-LLM, an encoder-free model that feeds lightly pre-processed Mel spectrogram patches directly into the LLM through a linear projection. This allows the LLM to learn speech-text alignment purely through its own parameters, without any dedicated audio encoder. The team conducted extensive experiments on automatic speech recognition (ASR) and text-to-speech (TTS) tasks, evaluating on the OpenASR leaderboard public sets and production-level scaling experiments.

For ASR, Mel-LLM achieves competitive performance with only limited degradation compared to encoder-initialized counterparts, especially when data is abundant. However, when data is limited, initialization from a multimodal checkpoint (specifically Phi-4-MM) proved crucial for maintaining performance. Ablation studies revealed which LLM layers are less relevant to speech encoding. For TTS, preliminary results using a next-token VAE approach demonstrate feasibility, though performance is not yet optimal. The work establishes the feasibility of a fully unified encoder-free architecture for autoregressive speech-text modeling, potentially simplifying future speech AI systems.