Email in the Era of LLMs

A team from the University of Chicago and the Allen Institute for AI has published a groundbreaking study titled 'Email in the Era of LLMs' on arXiv. To understand how large language models are reshaping professional communication, they created the HR Simulator—a game where players act as HR officers crafting emails for delicate workplace situations. Their analysis of over 600 emails, evaluated by LLMs acting as judges, reveals a stark performance gap: human-written emails achieved only a 23.5% success rate, while emails written by LLMs like GPT-4 scored between 48% and 54%. This suggests AI judges consistently favor AI-generated communication in these nuanced social tasks.

The research uncovers a critical trend: larger, more capable models show homogenized judgment and a clear preference for tact. Weaker models favored direct, less tactful strategies, whereas advanced models like GPT-4 consistently chose more diplomatic approaches. When humans and LLMs collaborated, the results were transformative, boosting success rates from 40% to nearly 100% in one scenario. However, the study also highlights a key limitation: while LLM rewrites successfully made human emails more formal and empathetic, they struggled to replicate the authentic, casual tone of low-empathy, low-formality human writing. This points to a fundamental challenge in current post-training methods for AI.