Netflix's MediaFM tri-modal embeddings personalize trailers and artwork

Netflix researchers (Emma Yanyang Kong and 7 co-authors) published a new arXiv paper detailing how multimodal embeddings reshaped production systems for asset personalization—the artwork images and video previews that drive content discovery. Traditional models relied on ID-based interaction history, leaving them blind to asset content and unable to serve newly launched titles. The team tackled this with two major systems. First, augmenting a two-tower model with pretrained CLIP image embeddings unlocks cross-title, cross-canvas knowledge transfer, letting a single model serve all five Netflix artwork canvas types and replacing five separately trained per-canvas models. A lightweight extension reuses CLIP's joint text-image space to make artwork personalization query-aware in search.

Second, they introduce MediaFM, an in-house tri-modal foundation model trained on a large-scale corpus of shots from Netflix's catalog, fusing visual (SeqCLIP), audio (wav2vec 2.0), and timed-text signals. For video preview personalization, MediaFM decisively beats strong visual-only baselines in both offline and online A/B tests. Crucially, the team developed a simple offline proxy task—predicting the popularity-based winner from embeddings alone—to rank embedding models and versions, pruning the choice space before any end-to-end integration or A/B test. This proxy now gates every new MediaFM checkpoint. The paper also shares production engineering decisions: shared embedding infrastructure, low-latency serving, and cheap screening, plus design tradeoffs and failure modes.