StrAD brings real-time audio descriptions to full-length videos

Audio description (AD) makes visual content accessible to blind and low-vision people, but manual creation is slow and costly, covering only a fraction of media. Existing automatic systems treat AD as short clip captioning, requiring timestamp annotations and character databases. StrAD, created by Julian Spravil, Sebastian Houben, and Sven Behnke, reframes AD as streaming dense video captioning. It slides a window over full-length videos and inserts descriptions directly into transcripts without ground-truth timestamps, supporting both fine-tuned models and zero-shot vision-language prompting. The accompanying benchmark spans diverse genres—films, documentaries, shorts, performances, and video games—making full-video AD evaluation measurable for the first time.

On segment-level tasks with given timestamps, StrAD-FT sets a new state of the art on CMD-AD with 36.3 CIDEr, a +10.0 improvement over Shot-by-shot, and stays competitive on MAD-Eval at 24.9 CIDEr. For the full-video streaming task, StrAD-FT reaches a SODA score of 2.4 versus 1.1 for the zero-shot baseline StrAD-Zero, though both show limitations in temporal localization and narrative coherence. While prior work used offline, multi-stage pipelines, StrAD is the first streaming approach that generates ADs on the fly. This progress is a prerequisite for scaling accessibility across long-form content, from feature films to gameplay.