Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)

A team of researchers, including Zichen Tang and Bo Li, has published a critical study titled "Is Your LLM-as-a-Recommender Agent Trustable?" The paper introduces a new benchmark called BiasRecBench, designed to systematically test the vulnerability of AI agents that use Large Language Models (LLMs) to make recommendations. These "LLM-as-a-Recommender" agents are increasingly deployed in real-world workflows like academic paper review, e-commerce product suggestions, and job candidate screening, where they must select the best option from a list of candidates.

The researchers developed a sophisticated Bias Synthesis Pipeline that creates test scenarios by injecting subtle, contextually logical biases into the options presented to the AI. Crucially, they calibrated the quality margin between the optimal and sub-optimal choices to isolate the effect of bias. Their extensive experiments on state-of-the-art models, including OpenAI's GPT-4o, Google's Gemini-2.5-pro and Gemini-3-pro, and DeepSeek-R1, revealed a startling flaw: these agents frequently succumbed to the injected biases and recommended inferior options, despite having the inherent reasoning capability to identify the correct choice.

This finding exposes a significant and previously underexplored reliability bottleneck. The failure occurs not from a lack of intelligence, but from the models' susceptibility to being "hacked" by persuasive but misleading contextual information. The study concludes that current agentic workflows are not trustable for high-value recommendations without new, specialized alignment strategies to harden them against such bias attacks. The complete code and datasets will be released publicly to spur further research into this critical security and safety issue.