VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

A research team from KAIST has introduced VLM2Rec, a novel framework designed to tackle a fundamental problem plaguing multimodal AI recommendation systems: modality collapse. When powerful Vision-Language Models (VLMs) like CLIP or BLIP are fine-tuned for tasks like sequential product recommendation, a common failure mode occurs. During optimization, the model's learning process becomes dominated by signals from a single modality—often text—while the other modality's representation quality degrades. This imbalance, where one data stream 'collapses' and loses its informative power, severely undermines the accuracy and robustness of the final recommendations, as critical visual information from product images is effectively ignored.

VLM2Rec proposes a two-pronged technical solution to enforce balanced learning. First, it employs Weak-modality Penalized Contrastive Learning, which dynamically identifies and applies stronger learning signals to the lagging modality during training, rectifying the gradient imbalance. Second, it uses Cross-Modal Relational Topology Regularization, a technique that preserves the geometric relationships and consistency between the image and text embedding spaces, ensuring they evolve together rather than diverging. Extensive experiments demonstrate that this approach allows VLMs to function effectively as high-capacity, collaborative-filtering-aware encoders.

The result is a system that consistently outperforms existing state-of-the-art baselines across diverse recommendation scenarios. By resolving modality collapse, VLM2Rec unlocks the full potential of VLMs for understanding complex, sequential user interactions with multimodal content, leading to significantly more accurate and reliable recommendations for platforms like e-commerce and streaming services.