RWTH Aachen's gradient method aligns speech for any ASR model without retraining

Speech-to-text alignment—pinpointing each word’s exact time in audio—is built into some ASR models (CTC, transducer) but absent in attention-based encoder-decoders (AED) and speech LLMs. Existing fixes rely on attention weights or encoder-frame grids, limiting temporal precision. A new paper from RWTH Aachen proposes a universal gradient-based approach: for any differentiable ASR model, compute the gradient of each teacher-forced token log probability with respect to the input audio, reduce it to per-frame saliency, then decode word boundaries via a single dynamic-programming pass. The method needs no training, no extra heads, and no model modification—it works on CTC, transducer, AED, and speech LLMs alike, and aligns on the raw input grid rather than the coarser encoder grid.

Evaluated on 16 models across four families (including speech LLMs) using TIMIT (read) and Buckeye (spontaneous) speech, the gradient approach produces usable alignment for every model. It generally trails strong native aligners (like CTC’s forced alignment) but outperforms them where native alignment is weak, particularly in streaming models. Its main drawback is computational cost: one backward pass per token. Still, the method offers a practical, zero-training alternative for obtaining word timings in any modern ASR system, opening the door to more accurate transcription analysis and audio event detection without architectural constraints.