ESPnet3 slashes training time 21 min, hits 80%+ GPU utilization
New open-source framework integrates datasets in just 46 lines of code
Speech and audio research has entered the foundation model era, but existing frameworks demand significant engineering effort to handle massive datasets and complex workflows. ESPnet3 addresses this with a modular system architecture built on configuration-driven dataset composition and unified Python-based workflows. At its core, the DataOrganizer abstraction enables flexible dataset integration, while dataset sharding cuts memory overhead for large-scale training. Researchers can also inject recipe-specific logic via lightweight stage overrides, keeping the framework extensible without bloat.
In benchmark tests on OWSM pre-training, ESPnet3 delivered tangible improvements: a 21.1-minute reduction in per-epoch training time compared to ESPnet2, and over 80% GPU utilization during multi-node training. For fine-tuning experiments, integrating new models and datasets required only about 46 lines of additional code—dramatically lowering the barrier for experimentation. The framework is accepted at Interspeech 2026 and will be publicly released with model checkpoints and training logs, making it a practical infrastructure for accelerating speech AI research.
- Per-epoch training time cut by 21.1 minutes vs ESPnet2 during OWSM pre-training
- Achieves over 80% GPU utilization in multi-node training scenarios
- New models and datasets can be integrated with approximately 46 lines of additional code
Why It Matters
Makes large-scale speech AI experiments vastly more efficient, reducing engineering overhead and accelerating foundation model research.