LALM matches fine-tuned audio LLMs using alignment-only training

Multimodal large language models (MLLMs) are usually built through a multi-stage pipeline: cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. That process assumes adapting an LLM to a new modality requires extensive task-specific supervision. But as pretrained LLMs grow more capable, researchers from Tencent AI Lab and other institutions wondered whether all that heavy lifting is necessary—or if pure alignment alone could transfer an LLM's existing reasoning and instruction-following skills to audio.

Their new model, LALM (Lab-Large Audio-Language Model), takes an extreme minimal-intervention approach. Both the audio encoder and the LLM remain completely frozen; only a lightweight projector is learned. They train on (audio, response) pairs built using Self-Generated Data Construction, where the LLM itself expands captions into free-form responses without any explicit task instructions. This borrows from the AzeroS framework and eliminates the need for hand-curated instruction datasets.

Results across four benchmarks—MMAU, MMAR, MMSU, and MMAU-Pro—show LALM matches or surpasses baselines that underwent SFT and preference tuning, while training on substantially less data. Because the LLM stays frozen, the model preserves its native instruction-following competence and can port seamlessly to newer LLM generations. The authors argue this reduces multimodal extension to a simple projector-training problem that generalizes across modalities and adapts rapidly to each LLM release, potentially reshaping how future audio-language models are built.