NAVER LABS speech pipeline hits COMET 0.781 with Qwen3-4B backbone
100k synthetic examples and SeamlessM4T-v2 encoder boost instruction-following speech translation.
NAVER LABS has released a re-implementation of their IWSLT 2025 instruction-following pipeline, adapted for the IWSLT 2026 Shared Task under constrained conditions (short audio track). The system uses Meta's SeamlessM4T-v2-large as the speech encoder and Alibaba's Qwen3-4B-Instruct as the LLM backbone—both mandated components for the task. The three-stage training methodology—projector alignment, text-only LoRA fine-tuning, and multimodal merging—remains unchanged from their award-winning 2025 design.
A key contribution is the creation of 100k synthetic instruction-following examples across ten speech-centric task types (10k per task), generated from the provided corpora. This synthetic data is suitable for Stage 3 fine-tuning. The primary model achieves COMET 0.781 on EN-ZH speech translation and BERTScore-F1 0.346 on English SQA (spoken question answering) on the MCIF benchmark. This work demonstrates how to effectively combine a speech encoder with a compact LLM for multimodal instruction-following tasks.
- Uses SeamlessM4T-v2-large as speech encoder and Qwen3-4B-Instruct as LLM backbone
- Generated 100k synthetic examples across 10 speech task types for fine-tuning
- Achieves COMET 0.781 on EN-ZH translation and BERTScore-F1 0.346 on English SQA
Why It Matters
Enables accurate speech-to-speech instruction following with a small, efficient model—useful for real-time voice assistants.