New Audio Captioning Model Uses AudioSet Keywords for Efficient Descriptions
A compact 6-layer BART decoder matches larger models on audio captioning benchmarks.
Automatic Audio Captioning (AAC) aims to translate complex acoustic scenes into natural language, but existing methods suffer from word-selection ambiguity and often depend on massive sequence-to-sequence or LLM-based architectures. This new paper from researchers at the Indian Institute of Technology (IIT) Hyderabad proposes a resource-efficient alternative that explicitly grounds caption generation in auxiliary semantics from AudioSet.
The framework uses a ConvNeXt encoder to extract frame-level acoustic representations, which are then augmented with the top-K predicted AudioSet keywords. These keywords provide structured contextual cues that guide a compact six-layer BART-style decoder, enabling high-quality caption generation without the computational overhead of LLM-scale models. Evaluations on the Clotho V2 and AudioCaps benchmarks show competitive caption quality under practical deployment constraints, demonstrating that a smaller, semantically grounded architecture can match or approach the performance of much larger systems while being far more efficient.
- Proposes a resource-efficient AAC framework using ConvNeXt encoder and top-K AudioSet keyword augmentation.
- Employs a compact six-layer BART-style decoder, avoiding LLM-scale parameter counts.
- Achieves competitive caption quality on Clotho V2 and AudioCaps benchmarks with lower computational cost.
Why It Matters
Enables practical, efficient audio captioning for deployment in resource-constrained environments without sacrificing quality.