New study reveals AI product recommendations collapse under minor phrasing changes
Minor rephrasings of the same buyer intent cause LLMs to flip brand recommendations with near-random consistency, exposing a fundamental flaw in the metrics powering today's AI-driven SEO industry.
A new paper from Will Jack and colleagues exposes a critical flaw in retrieval-augmented generation (RAG) for commercial product recommendations: paraphrase brittleness. Testing 6,000 natural paraphrases of the same buyer intent (e.g., 'best CRM' vs. 'top CRM for SaaS startups') against 6,000 same-prompt reruns on OpenAI and Anthropic models, the authors found that the Jaccard similarity between brand recommendation sets from two paraphrases was only 0.288 for cosmetic rewording and 0.135 for constraint-adding rewording. In contrast, same-prompt reruns achieved 0.50–0.61 similarity. The prompt string, not the underlying buyer intent, drives which brands appear.
This instability directly challenges the popular AEO/GEO (Answer Engine Optimization / Generative Engine Optimization) practice of tracking brand 'AI visibility' by counting mentions over a fixed set of prompts. The paper shows that the dominant variance in such tracking comes from which paraphrase the tracker issues, not the model's actual behavior toward the brand. Even increasing reasoning effort does not close the gap (bounded by ±0.05). While efficient multi-prompt evaluation methods exist in the literature, the natural buyer-phrasing space is vastly larger than the benchmark-scale sets those methods have been validated on. The authors conclude that prompt-by-prompt mention tracking is structurally unstable as a measurement unit, and meaningful improvement likely requires a different unit altogether.
- Same-intent paraphrases produce Jaccard similarities of only 0.135–0.288, meaning LLM brand recommendations are nearly random under minor phrasing changes.
- Current AEO tools from BrightEdge, SearchPilot, and MarketMuse rely on prompt stability that does not exist; their brand-tracking metrics may be measuring noise.
- The low rerun baseline (0.50–0.61) shows inherent LLM stochasticity even on identical prompts, demanding multi-prompt averaging for any reliable measurement.
Why It Matters
LLM-based recommendation systems are fundamentally brittle; the AEO industry must rebuild its metrics on statistical grounds.