VibeVoice-ASR-BitNet runs 2.3x faster than Whisper on edge CPUs

Microsoft researchers have released VibeVoice-ASR-BitNet, a compressed automatic speech recognition (ASR) model optimized for real-time inference on edge CPUs. The model employs heterogeneous quantization tailored to different stages of the pipeline: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To recover accuracy under such aggressive compression, the team applied a progressive quantization-aware training (QAT) strategy. For inference, custom SIMD kernels and fused operators were implemented within the ggml framework, targeting both ARM and x86 platforms.

VibeVoice-ASR-BitNet achieves real-time recognition with a real-time factor (RTF) below 1 using as few as 3 CPU threads. Compared to OpenAI's Whisper at similar model sizes (~1.6 GB), the new model is 1.6–2.3x faster, with only modest accuracy degradation relative to the FP16 baseline. This makes it highly suitable for on-device applications where latency and power are critical, such as voice assistants, transcription on mobile devices, and edge servers. The results demonstrate that combining stage-specific quantization with efficient inference kernels can bring server-grade ASR performance to resource-constrained hardware.