Mult-DPO: New method aligns LLM recommenders with multi-item user preferences

A team of researchers from Meta AI and the University of Virginia has proposed Mult-DPO (Multinomial Direct Preference Optimization), a new alignment technique for large language model (LLM)-based recommender systems. Traditional DPO, widely used for aligning LLMs with human preferences, relies on pairwise comparisons (e.g., item A over item B). However, in real-world recommendation scenarios, user feedback is rarely pairwise—instead, users often engage with multiple items simultaneously, creating set-wise preferences where every positive item should rank above every negative item, with no prescribed order among positives or among negatives.

To address this, the authors leverage the Plackett-Luce (PL) ranking model, which naturally extends the Bradley-Terry model (used in vanilla DPO) from pairwise to full rankings. However, adapting PL to set-wise preferences requires marginalizing over all possible orderings of positive items, a combinatorial problem. The key innovation of Mult-DPO is a tractable multinomial surrogate likelihood that operates on the same reward-induced weight space but avoids the combinatorial explosion. The researchers prove that the multinomial DPO loss is a tractable upper bound on the marginalized PL DPO loss, and they characterize the bound's tightness based on the relative total weight of positives versus negatives. This provides practical insights: using richer or harder negatives can tighten the bound and improve performance. The method is further extended to handle multiple preference levels (e.g., very liked, liked, neutral, disliked). The paper includes open-source code on GitHub, making it accessible for further experimentation and deployment in production recommender systems.