AttuneBench tests LLM emotional intelligence with real conversations

A team of researchers (Lubrano et al.) from multiple institutions released AttuneBench, a new benchmark for evaluating emotional intelligence in large language models. Unlike prior synthetic or single-turn tests, AttuneBench is grounded in 200 real, multi-turn conversations where humans chatted with anonymized LLMs and provided turn-by-turn annotations of their own emotional state, the model’s behavior, and their preferred response. The benchmark tests models across four dimensions: emotion recognition, behavioral classification, preference prediction, and judged response quality.

Across 11 evaluated models, the authors found that rankings on these four capabilities are largely independent, meaning emotional intelligence is not a single skill but a bundle of separable abilities. Notably, preference alignment and response quality were far more model-discriminating than simple emotion-label accuracy. The takeaway: being emotionally intelligent as an AI means predicting the kind of response a specific user wants in a specific conversational context—a nuance that aggregate scores and synthetic tests miss entirely.