PHA-Net's prototype alignment lifts video retrieval recall by 19.2%

A team of researchers (Xiaolun Jing et al.) has released PHA-Net (Prototype-based Hierarchical Alignment Network), a new approach to text-video retrieval that outperforms existing hierarchical alignment methods. The paper, posted on arXiv, targets the persistent problem of semantic mismatch between concise text queries and information-rich video content. While prior state-of-the-art models use multiple language-video attention modules to align features at individual, local, and global levels, they only optimize visual representations and incur prohibitive computational costs. PHA-Net instead introduces multiple modality-shared prototypes as a learnable bridge, jointly optimizing both text and video representations for cross-modal alignment.

The architecture also tackles the issue of imbalanced token semantics, where weak tokens undermine retrieval performance. Its prototype-supported token merge module enhances high-semantic tokens and suppresses weak ones using prototype guidance. Additionally, a prototype contrastive loss forces textual and visual prototypes from the same prototype to be more similar than those from different prototypes, ensuring each prototype captures distinct semantic information. The authors tested PHA-Net on four standard benchmarks and report significant gains in the sum of all recalls: 8.8% on MSR-VTT, 19.2% on ActivityNet, 0.7% on VATEX, and 4.9% on Charades. Code is available on arXiv, making it a strong new baseline for multimodal retrieval research.