OpenAI's GPT-4o and others beat sentiment analysis for social sign prediction

A team of researchers (Bombino, Boldrini, Passarella, Conti) from the Institute of Informatics and Telematics at CNR in Italy has published a paper on arXiv demonstrating a new methodology for annotating positive vs. negative user interactions to infer social relationship signs. Current approaches rely on sentiment analysis, which conflates the emotional tone of content with the relational nature of the exchange—for instance, sarcastic praise can be mislabeled as positive. The team instead leverages large language models (LLMs) in a zero-shot setting to directly identify interaction-level relational signals: personal praise directed at the interlocutor (positive tie) and personal attacks (negative tie).

They tested four models—OpenAI's GPT-4o and GPT-5.4-mini, Google's Gemma2:9b, and Alibaba's Qwen2.5:7b—across three prompt designs of increasing complexity, using two human-annotated datasets of 298 and 340 texts respectively. Results show zero-shot LLMs achieve good classification without any task-specific training. Attack detection proved robust to prompt design and model choice, while praise detection was more sensitive, reflecting the greater subjectivity of positive relational gestures. The findings lay the groundwork for integrating LLM-based relational annotation into social sign prediction pipelines, potentially improving network analysis, recommendation systems, and moderation tools.