When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis
Political debate agents often flip roles—Mistral holds stance 67% of the time vs Claude's 39%.
A new empirical study published on arXiv (2604.27228) by Juergen Dietrich delivers the first systematic test of role fidelity in multi-agent LLM pipelines used for democratic discourse analysis. The TRUST pipeline assigns adversarial roles—‘advocate’ and ‘legitimizer’—to different LLMs to generate structured multi-perspective assessments of political statements. The core assumption: models reliably maintain their assigned roles. Dietrich's epistemic stance classifier identifies roles from reasoning text without surface vocabulary, then measures fidelity across 60 political statements (30 English, 30 German) using four metrics: Role Drift Index (RDI), Expected Drift Distance (EDD), Directional Drift Index (DDI), and Entropy-based Role Stability (ERS).
The results reveal a stark performance gap: Mistral Large achieves 67% role fidelity versus Claude Sonnet's 39%—a 28 percentage-point difference. Two failure modes emerge: the Epistemic Floor Effect, where fact-check results create an absolute lower bound blocking the legitimizing role, and Role-Prior Conflict, where training-time knowledge overrides role instructions for factually unambiguous statements. Both stem from a single mechanism: Epistemic Role Override (ERO). Notably, Claude actively switches to the opposing stance, while Mistral abandons its role without reversal. Fact-check provider choice matters: Perplexity significantly reduces Claude's fidelity on German statements (Delta = -15pp, p = 0.007) but leaves Mistral unaffected. The paper concludes that multi-agent systems validated without role-fidelity measurement may systematically misrepresent the epistemic diversity they were designed to provide.
- Mistral Large maintains role fidelity at 67% vs. Claude Sonnet at 39% across 60 political statements.
- Two failure modes identified—Epistemic Floor Effect and Role-Prior Conflict—both caused by Epistemic Role Override (ERO).
- Perplexity’s fact-checking reduces Claude’s role fidelity on German statements by 15 percentage points; Mistral Large remains unaffected.
Why It Matters
Multi-agent LLM systems can distort democratic debate unless role fidelity is measured and validated.