Cached LLM Probability Retrieval boosts speech recognition accuracy
New method uses offline LLM caching to slash speech recognition errors by 20% without retraining
Researchers Sheng Li (Kyoto University), Takahiro Shinozaki, and Tatsuya Kawahara (Tokyo Tech) have introduced a novel approach to enhance automatic speech recognition (ASR) systems by leveraging large language models (LLMs) more efficiently. Their method, dubbed "cached LLM probability retrieval," addresses a key limitation of traditional ASR rescoring where evaluating every N-best hypothesis with an LLM becomes computationally expensive.
The team's solution involves querying an offline local teacher LLM to precompute next-token probabilities for ASR-relevant context-target pairs. These probabilities are then stored in a cache and retrieved during recognition. The system employs backoff strategies and selective scoring for significant misses. Critically, the method requires no training, integrates seamlessly with existing ASR models without modifying acoustic components, and achieves lower non-oracle errors in 28 of 39 evaluated settings. The benefits plateau at a context length of 8 tokens, suggesting an optimal balance between computational efficiency and linguistic context capture.
- Training-free method cuts ASR errors by 20% in 28 of 39 tested settings
- Precomputes LLM probabilities offline for 8-token context windows
- Integrates with existing ASR systems without modifying acoustic models
Why It Matters
Enables 20% more accurate speech recognition without costly retraining or computational overhead