RL + masked language models beat humans at ad headline generation

A new paper accepted at NAACL-HLT 2021 (Industry Track) tackles a persistent e-commerce challenge: generating high-quality ad headlines at scale. The team — Yashal Shakti Kanungo, Sumit Negi, and Aruna Rajan — introduces a programmatic solution that applies Reinforcement Learning (RL) policy gradient methods on top of a Transformer-based masked language model. Instead of treating each product in isolation, the model jointly conditions on multiple products a seller wants to advertise, creating cohesive, compelling headlines that pass creative quality bars.

Benchmark results show the method outperforms earlier Transformer and LSTM+RL approaches across overlap metrics (e.g., BLEU, ROUGE) and in human quality audits. Most strikingly, the model-generated headlines scored higher than actual human-submitted headlines in both grammatical correctness and creative appeal. This work demonstrates that RL fine-tuning of pre-trained language models can produce advertising copy that rivals — and even exceeds — human performance, a significant step toward fully automated, scalable ad creation for large e-commerce platforms.