vLLM pipeline unifies audio understanding and generation, preserves 80% CFG throughput
Open-source vLLM extension handles multi-stream audio tokens while cutting CFG overhead.
Large multimodal models excel at comprehension but struggle with generation, especially for audio. Speech-language models require generating multi-layered audio tokens via either decoupled autoregressive/non-autoregressive (AR+NAR) or synchronous multi-token prediction (MTP) with delay-pattern interleaving—both of which conflict with standard single-stream inference loops. A team led by Haoran Wang at CMU now presents a vLLM-based inference pipeline that bridges this gap, supporting both understanding and generation in a single, high-throughput framework.
The pipeline extends vLLM's autoregressive engine to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. A key contribution is overcoming the conventional wisdom that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, the implementation absorbs dual-request and logit-merging overheads, maintaining 80% of non-CFG throughput. The code is open-sourced, lowering the barrier for speech AI developers to build real-time, unified systems.
- Extends vLLM's autoregressive loop to handle delay-pattern de-interleaving and multi-stream sampling for audio tokens.
- Integrates an on-GPU acoustic decoder, enabling end-to-end waveform synthesis directly within the inference pipeline.
- Classifier-Free Guidance co-schedules paired conditional/unconditional requests in a continuous batch, achieving 80% of non-CFG throughput—debunking the 'CFG halves performance' myth.
Why It Matters
Reduces inference overhead for unified speech AI, enabling real-time audio understanding and generation in production systems.