Fine-tuned LLMs deliver stable, predictable ad recommendations at scale

Traditional ad recommendation systems optimize for accuracy metrics like recall or NDCG, but as ad inventory and liquidity grow with generative AI, prediction stability and predictability have become critical. Instability leads to repeatability problems, cold-start failures, and under-exploration, frustrating advertisers. This paper, presented at the SIGIR 2026 AgentSearch Workshop, introduces an evaluation framework for stability and predictability, along with a practical solution: a semantic candidate generation framework using fine-tuned LLMs.

The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the basis for graph-based expansion. This ensures that small creative variants from advertisers yield consistent and explainable delivery results. Tested in a large-scale industrial ad recommendation system, the framework demonstrated significant improvements in both offline and online A/B experiments, achieving gains in predictability while maintaining or improving traditional metrics. Although evaluated in ads, the authors note the framework can apply broadly to any large-scale recommendation and retrieval system facing scaling and predictability challenges.