Johns Hopkins & Amazon's StanceBench measures AI's stance detection from speech

A team of researchers from Johns Hopkins University and Amazon has introduced StanceBench, a novel benchmark designed to evaluate how well audio-capable large language models (LLMs) can detect interpersonal stances from conversational speech. Building on the Seamless Interaction corpus, the benchmark defines 9 specific stance dimensions—including empathy, politeness, warmth, assertiveness, honesty, attentiveness, and interaction-level cues like conflict regulation. StanceBench standardizes both single-speaker and interaction-based evaluations, and introduces LLM-as-a-judge metrics for robustness, bias, and inference accuracy. The work, accepted at Interspeech 2026, addresses a critical gap: while speech-to-speech dialogue models increasingly rely on prosody and social nuance, existing benchmarks largely ignore how well AI captures these subtle cues.

The benchmark's evaluation across the 9 stance dimensions reveals clear performance hierarchies. Empathy and politeness proved the easiest for LLMs to detect, likely due to their consistent acoustic markers. Warmth and assertiveness were moderately separable but showed positivity skew and asymmetry. The hardest dimension was honesty—LLMs struggled significantly, displaying high prompt order bias, suggesting that honest stance requires evidence across multiple conversational turns rather than a single utterance. Attentiveness was separable but aligned only weakly with human judgments. On interaction-level stances, such as conflict regulation, the benchmark found high context sensitivity and significant variance. These results underscore that today's audio LLMs are far from perfect at inferring nuanced social stances, and StanceBench provides a much-needed test bed for future improvements in conversational AI.