New Weighted Lookahead Scoring Restores Punctuation in Streaming ASR with 0.937 F1
Boosts punctuation accuracy to 93.7% F1 without adding latency to real-time speech.
A new paper from Sungmook Woo, Hyungu Kang, and Chanwoo Kim introduces an efficient punctuation restoration method specifically designed for streaming Automatic Speech Recognition (ASR) systems. Unlike generation-based approaches that suffer from latency and alignment failures under real-time constraints, this method uses a non-autoregressive scoring technique. It works by comparing punctuation insertion hypotheses against a no-insertion baseline at each word boundary, using a bounded lookahead of K subword tokens. A validation-calibrated threshold τ and weight α guide decisions without requiring parameter updates during inference.
On the IWSLT 2017 benchmark, the method achieves a 4-class macro F1 of 0.893 in the no-fine-tuning setting (K=2) and 0.937 after fine-tuning, significantly outperforming the prompt-based baseline (0.566) and a fine-tuned ELECTRA baseline (0.913) under the same lookahead budget. The researchers also conducted ablation studies on the lookahead budget K, revealing how smaller windows trade accuracy for lower latency. Accepted at IJCNN 2026, this work offers a practical solution for making real-time speech transcripts more readable without compromising streaming performance.
- Achieves macro F1 of 0.937 (fine-tuned) and 0.893 (zero-shot) on IWSLT 2017, beating prompt-based (0.566) and ELECTRA (0.913) baselines.
- Uses a non-autoregressive scoring method with bounded K-subword-token lookahead and no parameter updates during inference.
- Preserves original transcript and makes punctuation decisions per word boundary, eliminating latency issues of generation-based approaches.
Why It Matters
Makes real-time ASR transcripts more readable for professionals without adding latency, improving productivity in meetings, captions, and voice interfaces.