Interleaved Stacking Speeds Up Speech Model Distillation Training
New method preserves layer identity, cutting training time without accuracy loss.
Eungbeom Kim and Kyogu Lee have introduced interleaved stacking, a training acceleration technique for speech foundation model (SFM) distillation, accepted at Interspeech 2026. While distillation creates efficient student models for low-resource environments, the training process itself is slow. Their method progressively increases model depth during training—unlike prior stacking techniques that shuffle layer positions and cause performance drops. By consistently preserving each layer's original position, interleaved stacking maintains the specialized knowledge encoded in each SFM layer, leading to faster deployment without sacrificing accuracy.
The method was validated on the SUPERB benchmark, a standard for evaluating speech models on multiple tasks. Results show that interleaved stacking significantly reduces training time compared to existing distillation approaches, while achieving competitive or superior performance. This breakthrough addresses a key bottleneck in deploying compact speech models for real-world applications like voice assistants, transcription, and audio processing on edge devices. The paper is available on arXiv (2606.11766) and accepted at Interspeech 2026.
- Interleaved stacking preserves layer position consistency during progressive depth training, unlike existing stacking methods that cause performance degradation.
- The technique accelerates SFM distillation training, enabling faster deployment of efficient speech models for low-resource environments.
- Validated on the SUPERB benchmark and accepted at Interspeech 2026, showing competitive performance without quality loss.
Why It Matters
Faster, cheaper deployment of compact speech models for devices and real-time apps without sacrificing accuracy.